How deep neural networks can improve emotion recognition on video data

We consider the task of dimensional emotion recognition on video data using deep learning. While several previous methods have shown the benefits of training temporal neural network models such as recurrent neural networks (RNNs) on hand-crafted features, few works have considered combining convolutional neural networks (CNNs) with RNNs. In this work, we present a system that performs emotion recognition on video data using both CNNs and RNNs, and we also analyze how much each neural network component contributes to the system's overall performance. We present our findings on videos from the Audio/Visual+Emotion Challenge (AV+EC2015). In our experiments, we analyze the effects of several hyperparameters on overall performance while also achieving superior performance to the baseline and other competing methods.

• Publication year

  2016

• Venue

  International Conference on Information Photonics

• Publication date

  2016-02-24

• Fields of study

  Computer Science

• Identifiers
  DOI 10.1109/ICIP.2016.7532431arXiv 1602.07377
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• EmoNets: Multimodal deep learning approaches for emotion recognition in video

  2015cited by this paper
• Proceedings of the 2015 ACM on International Conference on Multimodal Interaction

  2015cited by this paper
• Multi-modal Dimensional Emotion Recognition using Recurrent Neural Networks

  2015cited by this paper
• Multimodal Affective Dimension Prediction Using Deep Bidirectional Long Short-Term Memory Recurrent Neural Networks

  2015cited by this paper
• Long Short Term Memory Recurrent Neural Network based Multimodal Dimensional Emotion Recognition

  2015cited by this paper
• AV + EC 2015 – The First Affect Recognition Challenge Bridging Across Audio, Video, and Physiological Data

  2015cited by this paper
• Prediction of asynchronous dimensional emotion ratings from audiovisual and physiological data

  2015cited by this paper
• Do Deep Neural Networks Learn Facial Action Units When Doing Expression Recognition?

  2015cited by this paper
• Recurrent Neural Networks for Emotion Recognition in Video

  2015cited by this paper
• A data-driven approach to cleaning large face datasets

  2014cited by this paper
• Facial Expression Recognition via a Boosted Deep Belief Network

  2014cited by this paper
• Combining modality specific deep neural networks for emotion recognition in video

  2013cited by this paper
• Introducing the RECOLA multimodal corpus of remote collaborative and affective interactions

  2013cited by this paper
• AU-aware Deep Networks for facial expression recognition

  2013cited by this paper
• ImageNet classification with deep convolutional neural networks

  2012cited by this paper
• Finding Celebrities in Billions of Web Images

  2012cited by this paper
• Learning active facial patches for expression analysis

  2012cited by this paper
• Induced Disgust , Happiness and Surprise : an Addition to the MMI Facial Expression Database

  2010cited by this paper
• The Extended Cohn-Kanade Dataset (CK+): A complete dataset for action unit and emotion-specified expression

  2010cited by this paper
• Facial expression recognition based on Local Binary Patterns: A comprehensive study

  2009cited by this paper
• Dlib-ml: A Machine Learning Toolkit

  2009cited by this paper
• PERSONALITY PROCESSES AND INDIVIDUAL DIFFERENCES Universals and Cultural Differences in the Judgments of Facial Expressions of Emotion

  2004cited by this paper
• Strong evidence for universals in facial expressions: a reply to Russell's mistaken critique.

  1994cited by this paper
• Evidence for a three-factor theory of emotions

  1977cited by this paper

• Comparison of Limitation in Facial Expression Recognition Based on Deep Learning and AI Techniques:Survey

  2025cites this paper
• Video-based Facial Expression Recognition of Students in Classroom Scenarios Using 3D Convolutional Neural Networks and Multi-Head Self-Attention

  2025cites this paper
• Game Analysis Using Multimodal Emotion Recognition

  2025cites this paper
• Res-RBG Facial Expression Recognition in Image Sequences Based on Dual Neural Networks

  2025cites this paper
• A Versatile Multimodal Learning Framework for Zero-Shot Emotion Recognition

  2024cites this paper
• Decoding Student Sentiments: Academic Emotion Prediction with CNN-ResNet

  2024cites this paper
• An Interaction Behavior Decision-Making Model of Service Robots for the Disabled Based on Human–Robot Empathy

  2024cites this paper
• Video Representation Learning for Conversational Facial Expression Recognition Guided by Multiple View Reconstruction

  2024cites this paper
• Predicting the Arousal and Valence Values of Emotional States Using Learned, Predesigned, and Deep Visual Features

  2024influential citation
• Neural style transfer generative adversarial network (NST-GAN) for facial expression recognition

  2023cites this paper
• Sensory Maze: A camera-based remote expression therapy interface

  2023cites this paper
• Enhanced spatio-temporal 3D CNN for facial expression classification in videos

  2023cites this paper
• Prediction of Continuous Emotional Measures through Physiological and Visual Data

  2023cites this paper
• Facial emotion recognition based on deep transfer learning approach

  2023cites this paper
• Measuring Emotion Intensity: Evaluating Resemblance in Neural Network Facial Animation Controllers and Facial Emotion Corpora

  2023cites this paper
• Measuring Emotion Velocity for Resemblance in Neural Network Facial Animation Controllers and Their Emotion Corpora

  2023cites this paper
• Movements Recognition in the Human Body Based on Deep Learning Strategies

  2023cites this paper
• Emotion Classification from Speech and Text in Videos Using a Multimodal Approach

  2022cites this paper
• Four-Class Emotion Classification Problem using Deep Learning Classifiers

  2022cites this paper
• Survey on Emotion Recognition System

  2022cites this paper
• Meme Detection For Sentiment Analysis and Human Robot Interactions Using Multiple Modes

  2022cites this paper
• Facial identity protection using deep learning technologies: an application in affective computing

  2022influential citation
• Spatial Temporal Variation Graph Convolutional Networks (STV-GCN) for Skeleton-Based Emotional Action Recognition

  2021cites this paper
• Exploring Multi-Modality in Animal-Centered Computing

  2021cites this paper
• AttendAffectNet–Emotion Prediction of Movie Viewers Using Multimodal Fusion with Self-Attention

  2021cites this paper
• Towards Privacy-Preserving Affect Recognition: A Two-Level Deep Learning Architecture

  2021cites this paper
• Zero-shot Video Emotion Recognition via Multimodal Protagonist-aware Transformer Network

  2021cites this paper
• A visual self-attention network for facial expression recognition

  2021cites this paper
• Emotion Analysis using Facial Expressions in Video

  2021cites this paper
• Analysing Affective Behavior in the second ABAW2 Competition

  2021cites this paper
• Activity & Emotion Detection of Recognized kids in CCTV Video for Day Care Using SlowFast & CNN

  2021cites this paper
• Two-Way Feature Extraction Using Sequential and Multimodal Approach for Hateful Meme Classification

  2021cites this paper
• CNN-LSTM-Based Facial Expression Recognition

  2021cites this paper
• Convolutional Neural Networks-An Extensive arena of Deep Learning. A Comprehensive Study

  2021cites this paper
• A Survey on Infant Emotion Recognition through Video Clips

  2021cites this paper
• A Two-Stage Spatiotemporal Attention Convolution Network for Continuous Dimensional Emotion Recognition From Facial Video

  2021cites this paper
• Graph-Based Facial Affect Analysis: A Review

  2021cites this paper
• Enhanced convolutional LSTM with spatial and temporal skip connections and temporal gates for facial expression recognition from video

  2021influential citation
• Audio-video Emotion Recognition in the Wild using Deep Hybrid Networks

  2020cites this paper
• Facial Expression Recognition by Jointly Partial Image and Deep Metric Learning

  2020cites this paper
• Computer Vision and Image Processing: 4th International Conference, CVIP 2019, Jaipur, India, September 27–29, 2019, Revised Selected Papers, Part II

  2020cites this paper
• Multi-Modal Recurrent Attention Networks for Facial Expression Recognition

  2020influential citation
• Group based emotion recognition from video sequence with hybrid optimization based recurrent fuzzy neural network

  2020cites this paper
• Personalized models for facial emotion recognition through transfer learning

  2020cites this paper
• CNN and LSTM-Based Emotion Charting Using Physiological Signals

  2020cites this paper
• Hybrid-Deep Learning Model for Emotion Recognition Using Facial Expressions

  2020cites this paper
• AutoFER: PCA and PSO based automatic facial emotion recognition

  2020cites this paper
• Automatic facial recognition using VGG16 based transfer learning model

  2020cites this paper
• Video Duygu Analizi

  2020cites this paper
• Video-EEG Based Collaborative Emotion Recognition Using LSTM and Information-Attention

  2020cites this paper
• Facial expression recognition based on bidirectional gated recurrent units within deep residual network

  2020cites this paper
• Deep Emotion Recognition through Upper Body Movements and Facial Expression

  2020cites this paper
• Exploiting Multi-CNN Features in CNN-RNN Based Dimensional Emotion Recognition on the OMG in-the-Wild Dataset

  2019cites this paper
• Deep Hybrid Neural Networks for Facial Expression Classification

  2019cites this paper
• Semantic processors of Multicore series for analysis of human emotional condition

  2019cites this paper
• Time-Continuous Emotion Recognition Using Spectrogram Based CNN-RNN Modelling

  2019cites this paper
• Tri-modal Recurrent Attention Networks for Emotion Recognition

  2019influential citation
• Attending to Emotional Narratives

  2019cites this paper
• Learning Hierarchical Emotion Context for Continuous Dimensional Emotion Recognition From Video Sequences

  2019influential citation
• Extended deep neural network for facial emotion recognition

  2019cites this paper
• Deep Learning Approach: Emotion Recognition from Human Body Movements

  2019cites this paper
• Multi-Objective Based Spatio-Temporal Feature Representation Learning Robust to Expression Intensity Variations for Facial Expression Recognition

  2019cites this paper
• Deep convolutional BiLSTM fusion network for facial expression recognition

  2019cites this paper
• Deep Learning for Human Affect Recognition: Insights and New Developments

  2019cites this paper
• Deep Learning for Face Expressions Detection: Enhanced Recurrent Neural Network with Long Short Term Memory

  2019cites this paper
• Estimation of affective dimensions using CNN-based features of audiovisual data

  2019cites this paper
• Interpretable Deep Neural Networks for Facial Expression and Dimensional Emotion Recognition in-the-wild

  2019cites this paper
• Continuous Emotion Recognition in Videos by Fusing Facial Expression, Head Pose and Eye Gaze

  2019influential citation
• Affective Computing for Large-scale Heterogeneous Multimedia Data

  2019cites this paper
• Dynamic Facial Models for Video-Based Dimensional Affect Estimation

  2019cites this paper
• Emotion Prediction in Movies Using Visual Features & Genre Information

  2019cites this paper
• A Hybrid Facial Expression Recognition System Based on Recurrent Neural Network

  2019cites this paper
• Modeling Emotion in Complex Stories: The Stanford Emotional Narratives Dataset

  2019cites this paper
• Facial-Expression Recognition from Video using Enhanced Convolutional LSTM

  2019cites this paper
• InSocialNet: Interactive visual analytics for role—event videos

  2019cites this paper
• Facial Expression Recognition Using a Large Out-of-Context Dataset

  2018cites this paper
• Emotion Recognition Using Multimodal Approach

  2018cites this paper
• A Deep-Learning Based Model for Emotional Evaluation of Video Clips

  2018cites this paper
• A Compact Deep Learning Model for Robust Facial Expression Recognition

  2018cites this paper
• Hybrid deep neural networks for face emotion recognition

  2018cites this paper
• Video analysis for augmented cataract surgery

  2018cites this paper
• End-to-End Continuous Emotion Recognition from Video Using 3D Convlstm Networks

  2018cites this paper
• Spatiotemporal Attention Based Deep Neural Networks for Emotion Recognition

  2018influential citation
• Survey of Approaches for Building Emotion Detection Applications using a Multi-modal Approach

  2018cites this paper
• Review of Automatic Emotion Recognition Through Facial Expression Analysis

  2018cites this paper
• Multimodal Data Fusion and Prediction of Emotional Dimensions Using Deep Neural Network

  2018cites this paper
• Learning deep facial expression features from image and optical flow sequences using 3D CNN

  2018cites this paper
• Enhanced speech emotion detection using deep neural networks

  2018cites this paper
• Monitoring tool usage in surgery videos using boosted convolutional and recurrent neural networks

  2018cites this paper
• Audio-Visual Attention Networks for Emotion Recognition

  2018influential citation
• Multimodal Fusion of Spatial-Temporal Features for Emotion Recognition in the Wild

  2017cites this paper
• End-to-end learning for dimensional emotion recognition from physiological signals

  2017cites this paper
• Multimodal Prediction of Affective Dimensions via Fusing Multiple Regression Techniques

  2017cites this paper
• Audio-visual emotion recognition using deep transfer learning and multiple temporal models

  2017cites this paper
• Statistical Selection of CNN-based Audiovisual Features for Instantaneous Estimation of Human Emotional States

  2017cites this paper
• Taking the Hidden Route: Deep Mapping of Affect via 3D Neural Networks

  2017cites this paper
• Robust emotion recognition from low quality and low bit rate video: A deep learning approach

  2017influential citation
• Modeling multimodal cues in a deep learning-based framework for emotion recognition in the wild

  2017cites this paper
• Recognition of Affect in the Wild Using Deep Neural Networks

  2017cites this paper
• Monitoring tool usage in cataract surgery videos using boosted convolutional and recurrent neural networks

  2017influential citation