Group-Level Emotion Recognition Using Hybrid Deep Models Based on Faces, Scenes, Skeletons and Visual Attentions

This paper presents a hybrid deep learning network submitted to the 6th Emotion Recognition in the Wild (EmotiW 2018) Grand Challenge [9], in the category of group-level emotion recognition. Advanced deep learning models trained individually on faces, scenes, skeletons and salient regions using visual attention mechanisms are fused to classify the emotion of a group of people in an image as positive, neutral or negative. Experimental results show that the proposed hybrid network achieves 78.98% and 68.08% classification accuracy on the validation and testing sets, respectively. These results outperform the baseline of 64% and 61%, and achieved the first place in the challenge.

• Publication year

  2018

• Venue

  International Conference on Multimodal Interaction

• Publication date

  2018-10-02

• Fields of study

  Computer Science

• Identifiers
  DOI 10.1145/3242969.3264990
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• EmotiW 2018: Audio-Video, Student Engagement and Group-Level Affect Prediction

  2018cited by this paper
• Smile Detection in the Wild Based on Transfer Learning

  2018cited by this paper
• Group emotion recognition with individual facial emotion CNNs and global image based CNNs

  2017cited by this paper
• Hand Keypoint Detection in Single Images Using Multiview Bootstrapping

  2017cited by this paper
• Residual Attention Network for Image Classification

  2017cited by this paper
• VGGFace2: A Dataset for Recognising Faces across Pose and Age

  2017cited by this paper
• Bottom-Up and Top-Down Attention for Image Captioning and VQA

  2017cited by this paper
• Squeeze-and-Excitation Networks

  2017cited by this paper
• Group-level emotion recognition using deep models on image scene, faces, and skeletons

  2017influential reference
• Convolutional Pose Machines

  2016cited by this paper
• Large-Margin Softmax Loss for Convolutional Neural Networks

  2016cited by this paper
• MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognition

  2016cited by this paper
• Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks

  2016cited by this paper
• Happiness level prediction with sequential inputs via multiple regressions

  2016cited by this paper
• Realtime Multi-person 2D Pose Estimation Using Part Affinity Fields

  2016cited by this paper
• Knowing When to Look: Adaptive Attention via a Visual Sentinel for Image Captioning

  2016cited by this paper
• Self-Critical Sequence Training for Image Captioning

  2016cited by this paper
• Rethinking the Inception Architecture for Computer Vision

  2015cited by this paper
• Deep Residual Learning for Image Recognition

  2015cited by this paper
• Group-level arousal and valence recognition in static images: Face, body and context

  2015cited by this paper
• Riesz-based Volume Local Binary Pattern and A Novel Group Expression Model for Group Happiness Intensity Analysis

  2015cited by this paper
• Deep Face Recognition

  2015cited by this paper
• The more the merrier: Analysing the affect of a group of people in images

  2015influential reference
• Stacked Attention Networks for Image Question Answering

  2015cited by this paper
• Automatic Group Happiness Intensity Analysis

  2015cited by this paper
• Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering

  2015cited by this paper
• Event detection: Ultra large-scale clustering of facial expressions

  2015cited by this paper
• Recurrent Models of Visual Attention

  2014cited by this paper
• Very Deep Convolutional Networks for Large-Scale Image Recognition

  2014cited by this paper
• Challenges in representation learning: A report on three machine learning contests

  2013cited by this paper
• ImageNet classification with deep convolutional neural networks

  2012cited by this paper
• Social functionality of human emotion.

  2012cited by this paper
• CENTRIST: A Visual Descriptor for Scene Categorization

  2011cited by this paper
• Facial Expression Based Automatic Album Creation

  2010cited by this paper
• Toward Practical Smile Detection

  2009cited by this paper
• ImageNet: A large-scale hierarchical image database

  2009cited by this paper
• 'Affective' computing and emotion recognition systems: the future of biometric surveillance?

  2005cited by this paper
• Perceiving affect from arm movement.

  2001cited by this paper
• Long Short-Term Memory

  1997cited by this paper

• Adaptive Key Role Guided Hierarchical Relation Inference for Enhanced Group-Level Emotion Recognition

  2026cites this paper
• Incorporating Scene Context and Semantic Labels for Enhanced Group-level Emotion Recognition

  2025cites this paper
• BAH Dataset for Ambivalence/Hesitancy Recognition in Videos for Behavioural Change

  2025cites this paper
• BAH Dataset for Ambivalence/Hesitancy Recognition in Videos for Digital Behavioural Change

  2025cites this paper
• Joint Multi-Scale Contextual and Noise Suppression for Group Emotion Recognition

  2025cites this paper
• A Hybrid Fusion Model for Group-Level Emotion Recognition in Complex Scenarios

  2025cites this paper
• Audiovisual Affect Recognition for Autonomous Vehicles: Applications and Future Agendas

  2024cites this paper
• Group-Level Emotion Recognition Using Hierarchical Dual-Branch Cross Transformer with Semi-Supervised Learning

  2024influential citation
• Implementing the Affective Mechanism for Group Emotion Recognition With a New Graph Convolutional Network Architecture

  2024cites this paper
• A Survey of Deep Learning for Group-level Emotion Recognition

  2024influential citation
• Automatic Emotion Recognition for Groups: A Review

  2023influential citation
• Social Event Context and Affect Prediction in Group Videos

  2023cites this paper
• A recent survey on perceived group sentiment analysis

  2023cites this paper
• Towards a Robust Group-Level Emotion Recognition via Uncertainty-Aware Learning

  2023cites this paper
• Multimodal Group Emotion Recognition In-the-wild Using Privacy-Compliant Features

  2023cites this paper
• Spatial-temporal Consistency based Crowd Emotion Recognition

  2023cites this paper
• Versatile Audio-Visual Learning for Emotion Recognition

  2023cites this paper
• Moving Beyond Benchmarks and Competitions: Towards Addressing Social Media Challenges in an Educational Context

  2023cites this paper
• A Self-Fusion Network Based on Contrastive Learning for Group Emotion Recognition

  2023cites this paper
• The landscape of facial processing applications in the context of the European AI Act and the development of trustworthy systems

  2022cites this paper
• Group Emotion Detection Based on Social Robot Perception

  2022influential citation
• Automatic Recognition of Emotional Subgroups in Images

  2022cites this paper
• Regional Attention Networks with Context-aware Fusion for Group Emotion Recognition

  2021influential citation
• Affective Image Content Analysis: Two Decades Review and New Perspectives

  2021cites this paper
• Challenging social media threats using collective well-being-aware recommendation algorithms and an educational virtual companion

  2021cites this paper
• Group-Level Emotion Recognition Using a Unimodal Privacy-Safe Non-Individual Approach

  2020cites this paper
• Revisiting crowd behaviour analysis through deep learning: Taxonomy, anomaly detection, crowd emotions, datasets, opportunities and prospects

  2020cites this paper
• Joint Prediction of Group-Level Emotion and Cohesiveness with Multi-Task Loss

  2020cites this paper
• Latest trends in emotion recognition methods: case study on emotiw challenge

  2020cites this paper
• Facial expression recognition in the wild, by fusion of deep learnt and hand-crafted features

  2020cites this paper
• Kreative Künstliche Intelligenz und Emotionen

  2020cites this paper
• Implicit Knowledge Injectable Cross Attention Audiovisual Model for Group Emotion Recognition

  2020cites this paper
• Social Relationship Recognition Based on A Hybrid Deep Neural Network

  2019cites this paper
• Automatic detection of students’ affective states in classroom environment using hybrid convolutional neural networks

  2019cites this paper
• EmotiW 2019: Automatic Emotion, Engagement and Cohesion Prediction Tasks

  2019cites this paper
• Group-level Cohesion Prediction using Deep Learning Models with A Multi-stream Hybrid Network

  2019cites this paper
• Automatic Group Cohesiveness Detection With Multi-modal Features

  2019cites this paper
• Graph Neural Networks for Image Understanding Based on Multiple Cues: Group Emotion Recognition and Event Recognition as Use Cases

  2019cites this paper
• Group Emotion Recognition in Adverse Face Detection

  2019cites this paper
• EmotiW 2018: Audio-Video, Student Engagement and Group-Level Affect Prediction

  2018influential citation
• Ordinal Regression Using Noisy Pairwise Comparisons for Body Mass Index Range Estimation

  2018cites this paper
• A Multimodal LSTM for Predicting Listener Empathic Responses Over Time

  2018cites this paper