Recognition of emotional states using voice, face image and thermal image of face

A new integration method is presented to recognize the emotional expressions. We attempted to use both voices and facial expressions. For voices, we use such prosodic parameters as pitch signals, energy, and their derivatives, which are trained by Hidden Markov Model (HMM) for recognition. For facial expressions, we use feature parameters from thermal images in addition to visible images, which are trained by neural networks (NN) for recognition. The thermal images are observed by infrared ray which is not influenced by lighting conditions. The total recognition rates show better performance than each performance rate obtained from isolated experiment. The results are compared with the recognition by human questionnaire.

• Publication year

  2000

• Venue

  Interspeech

• Publication date

  2000-10-16

• Fields of study

  Computer Science, Engineering

• Identifiers
  DOI 10.21437/ICSLP.2000-162
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• A method for detecting transitions of emotional states using a thermal facial image based on a synthesis of facial expressions

  2000cited by this paper
• Prosody and speech recognition

  1990cited by this paper

• Development of Machine-Learning-Based Facial Thermal Image Analysis for Dynamic Emotion Sensing

  2025cites this paper
• Audio-visual expression-based emotion recognition model for neglected people in real-time: a late-fusion approach

  2024cites this paper
• Improving Thermal Facial Emotion Recognition in Subject-Independent Scenarios Through Multigraph Convolutional Networks

  2024cites this paper
• Thermal heat distribution features for hand identification

  2022cites this paper
• Deep Learning for Audio Visual Emotion Recognition

  2022cites this paper
• Deep learning based multimodal emotion recognition using model-level fusion of audio-visual modalities

  2022cites this paper
• Feature Fusion In Multimodal Emotion Recognition System For Enhancement Of Human-Machine Interaction

  2021cites this paper
• Emotion recognition using multimodal matchmap fusion and multi-task learning

  2021cites this paper
• Human-Computer Communication Using Recognition and Synthesis of Facial Expression

  2021cites this paper
• Improving Ultrasound Tongue Image Reconstruction from Lip Images Using Self-supervised Learning and Attention Mechanism

  2021cites this paper
• MedSeq2Seq: A Medical Knowledge Enriched Sequence to Sequence Learning Model for COVID-19 Diagnosis

  2021influential citation
• A Quantum-Like multimodal network framework for modeling interaction dynamics in multiparty conversational sentiment analysis

  2020cites this paper
• The Use of AI for Thermal Emotion Recognition: A Review of Problems and Limitations in Standard Design and Data

  2020influential citation
• Unpaired Multimodal Facial Expression Recognition

  2020cites this paper
• A Multimodal Facial Emotion Recognition Framework through the Fusion of Speech with Visible and Infrared Images

  2020cites this paper
• Face Recognition and Its Multiple Applications

  2020cites this paper
• Weighted Normalization Fusion Approach For Multimodal Emotion Recognition System

  2020cites this paper
• SeLF: A Deep Neural Network Based Multimodal Sequential Late Fusion Approach for Human Emotion Recognition

  2019cites this paper
• Classroom engagement evaluation using computer vision techniques

  2019cites this paper
• A survey of emotion recognition methods with emphasis on E-Learning environments

  2019cites this paper
• An integrated approach to emotion recognition and gender classification

  2019cites this paper
• A Survey on: Facial Emotion Recognition Invariant to Pose, Illumination and Age

  2019cites this paper
• Illumination Resisted Approaches for Emotion Recognition Systems

  2019influential citation
• Comparative Analysis of Emotion Detection from Facial Expressions and Voice Using Local Binary Patterns and Markov Models: Computer Vision and Facial Recognition

  2018cites this paper
• Facial Expression Recognition Enhanced by Thermal Images through Adversarial Learning

  2018cites this paper
• A Portable Personality Recognizer Based on Affective State Classification Using Spectral Fusion of Features

  2018cites this paper
• Thermal Augmented Expression Recognition

  2018cites this paper
• Recognizing Emotional State of the Human Using Facial and Acoustic Features

  2018cites this paper
• Multimodal user state and trait recognition: an overview

  2018cites this paper
• Stress Recognition from Heterogeneous Data. (Reconnaissance de stress à partir de données hétérogènes)

  2017cites this paper
• Toward Use of Facial Thermal Features in Dynamic Assessment of Affect and Arousal Level

  2017cites this paper
• Multi-sensor fusion in body sensor networks: State-of-the-art and research challenges

  2017cites this paper
• Acquisition of Affect

  2017cites this paper
• A R EVIEW ON T EXTURE B ASED E MOTION R ECOGNITION FROM F ACIAL E XPRESSION

  2016cites this paper
• Stress Recognition from Heterogeneous Data

  2016cites this paper
• USING PLAYER'S FACIAL EMOTIONAL EXPRESSIONS AS A GAME INPUT

  2016cites this paper
• Time-Delay Neural Network for Continuous Emotional Dimension Prediction From Facial Expression Sequences

  2016cites this paper
• Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-Related Applications

  2016cites this paper
• A Survey of Autonomous Human Affect Detection Methods for Social Robots Engaged in Natural HRI

  2016cites this paper
• Facial Expression Recognition with Deep two-view Support Vector Machine

  2016cites this paper
• A multimodal emotion recognition system from video

  2016cites this paper
• Survey on Feature Extraction and Applications of Biosignals

  2016cites this paper
• Human-Robot Interactions for Single Robots and Multi-Robot Teams

  2016cites this paper
• A thermal emotion classifier for improved human-robot interaction

  2016cites this paper
• Real-time speech emotion recognition by minimum number of features

  2016cites this paper
• Characterization and recognition of mixed emotional expressions in thermal face image

  2016cites this paper
• Analyzing Contextual Polarity of Unstructured Data for Measuring Subjective Well-Being

  2016cites this paper
• Affective computing in the area of autism

  2015cites this paper
• Facial Expression Recognition Using Thermal Image Processing and Efficient Preparation of Training-data

  2015cites this paper
• Tensor Analysis and Fusion of Multimodal Brain Images This paper presents a general framework for tensor analysis of single-modality model inversion and multimodal data fusion using neuroimaging data as an example.

  2015cites this paper
• Decision-making content of an agent affected by emotional feedback provided by capture of human’s emotions through a Bimodal System

  2015cites this paper
• Tensor Analysis and Fusion of Multimodal Brain Images

  2015cites this paper
• HUMAN EMOTION RECOGNITION USING THERMAL IMAGE PROCESSING

  2015cites this paper
• Introduction to Emotion Recognition

  2015cites this paper
• Facial Changes during the Pronunciation of Velar and Palatal Consonants in IR Images

  2015cites this paper
• Expression Recognition from Visible Images with the Help of Thermal Images

  2015influential citation
• Affective State Level Recognition in Naturalistic Facial and Vocal Expressions

  2014cites this paper
• An interdisciplinary taxonomy of social cues and signals in the service of engineering robotic social intelligence

  2014cites this paper
• Thermal Face Recognition in Unconstrained Environments Using Histograms of LBP Features

  2014cites this paper
• Technologien für eine zustandsbasierte Suche im Internet der Dinge

  2014cites this paper
• Classifying emotions in rehabilitation robotics based on facial skin temperature

  2014cites this paper
• Human Expression Recognition using Facial Features

  2014cites this paper
• Method of Facial Expression Analysis Using Video Phone and Thermal Image

  2014cites this paper
• Fusion of Visible Images and Thermal Image Sequences for Automated Facial Emotion Estimation

  2014cites this paper
• Fusion of visible and thermal images for facial expression recognition

  2014influential citation
• Human Emotion Estimation Using Wavelet Transform and t-ROIs for Fusion of Visible Images and Thermal Image Sequences

  2014cites this paper
• Experimental verification on ocular features variation in terms of emotion stimuli and gender

  2014cites this paper
• Keynote Speaker-4: Artificial Intelligence and Emotion Actuator Design

  2014cites this paper
• Facial expression recognition of a speaker using thermal image processing and reject criteria in feature vector space

  2014cites this paper
• Recognition of emotions from video using acoustic and facial features

  2013cites this paper
• Embedded Design of an Emotion-Aware Music Player

  2013cites this paper
• A Multimodal Emotion Detection System during Human-Robot Interaction

  2013cites this paper
• Fusión sensorial para una plataforma de seguimiento

  2013cites this paper
• Categorical and dimensional affect analysis in continuous input: Current trends and future directions

  2013cites this paper
• Natural, Affect Aware interfaces: Gesture and Body Expressivity Aspects

  2013cites this paper
• Classification of speech under stress based on physical characteristics of vocal folds vibration

  2013cites this paper
• Robust facial expression recognition of a speaker using thermal image processing and updating of fundamental training data

  2013cites this paper
• Using non-verbal cues to (automatically) assess children's performance difficulties with arithmetic problems

  2013cites this paper
• GABRIEL ENRIQUE HERMOSILLA VIGNEAU

  2012cites this paper
• A system for facial expression recognition of a speaker using front-view face judgment, vowel judgment, and thermal image processing

  2012cites this paper
• Plate-forme multimodale pour la reconnaissance d'émotions via l'analyse de signaux physiologiques : Application à la simulation d'entretiens d'embauche. (Multimodal paltform for emotion recognition based on physiological responses : application in the interviews simulation)

  2012cites this paper
• A comparative study of thermal face recognition methods in unconstrained environments

  2012cites this paper
• Spontaneous Facial Expression Recognition by Fusing Thermal Infrared and Visible Images

  2012cites this paper
• Keynote Speaker-2: Emotion, Future Smart Technology, and Convergence

  2012cites this paper
• A stress-awareness approach to safer decision making in the trading process integrating biometric sensor technology

  2012cites this paper
• Facial Expression Recognition of Speaker Using Vowel Judgment and Features of Thermal Face Image

  2012cites this paper
• Spontaneous facial expression recognition by using feature-level fusion of visible and thermal infrared images

  2011cites this paper
• Novel emotion dynamic express for robot

  2011cites this paper
• An Effective Research of Emotion

  2011cites this paper
• Theories, methods and current research on emotions in library and information science, information retrieval and human-computer interaction

  2011cites this paper
• Exploring Fusion Methods for Multimodal Emotion Recognition with Missing Data

  2011cites this paper
• Facial expression recognition of a speaker using front-view face judgment, vowel judgment, and thermal image processing

  2011cites this paper
• Emotion dynamic express by fuzzy function for emotion robot

  2011cites this paper
• Facial expression recognition of a speaker using vowel judgment and thermal image processing

  2011cites this paper
• Significant Accomplishments, New Challenges, and New Perspectives

  2011cites this paper
• Vowel Judgment for Facial Expression Recognition of a Speaker

  2011cites this paper
• Towards Computer-Assisted Regulation of Emotions

  2011cites this paper
• Classification on Speech Emotion Recognition - A Comparative Study

  2010cites this paper
• SVM-MLP-PNN Classifiers on Speech Emotion Recognition Field - A Comparative Study

  2010cites this paper
• Facial expression recognition for speaker using thermal image processing and speech recognition system

  2010cites this paper