Tensor Fusion Network for Multimodal Sentiment Analysis

Multimodal sentiment analysis is an increasingly popular research area, which extends the conventional language-based definition of sentiment analysis to a multimodal setup where other relevant modalities accompany language. In this paper, we pose the problem of multimodal sentiment analysis as modeling intra-modality and inter-modality dynamics. We introduce a novel model, termed Tensor Fusion Networks, which learns both such dynamics end-to-end. The proposed approach is tailored for the volatile nature of spoken language in online videos as well as accompanying gestures and voice. In the experiments, our model outperforms state-of-the-art approaches for both multimodal and unimodal sentiment analysis.

• Publication year

  2017

• Venue

  Conference on Empirical Methods in Natural Language Processing

• Publication date

  2017-07-01

• Fields of study

  Computer Science

• Identifiers
  DOI 10.18653/v1/D17-1115arXiv 1707.07250
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Joint Robust Voicing Detection and Pitch Estimation Based on Residual Harmonics

  2019cited by this paper
• Opinion Mining and Sentiment Analysis

  2018cited by this paper
• A review of affective computing: From unimodal analysis to multimodal fusion

  2017cited by this paper
• A Practical Guide to Sentiment Analysis

  2017cited by this paper
• Combating Human Trafficking with Multimodal Deep Models

  2017cited by this paper
• Representation Learning for Speech Emotion Recognition

  2016cited by this paper
• OpenFace: An open source facial behavior analysis toolkit

  2016cited by this paper
• Speech emotion recognition using convolutional and Recurrent Neural Networks

  2016cited by this paper
• SenticNet 4: A Semantic Resource for Sentiment Analysis Based on Conceptual Primitives

  2016cited by this paper
• A Shared Task on Multimodal Machine Translation and Crosslingual Image Description

  2016cited by this paper
• Convolutional Experts Constrained Local Model for Facial Landmark Detection

  2016cited by this paper
• Image Captioning with Semantic Attention

  2016cited by this paper
• AVEC 2016: Depression, Mood, and Emotion Recognition Workshop and Challenge

  2016cited by this paper
• MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos

  2016influential reference
• EmoReact: a multimodal approach and dataset for recognizing emotional responses in children

  2016cited by this paper
• Select-Additive Learning: Improving Cross-individual Generalization in Multimodal Sentiment Analysis

  2016influential reference
• Convolutional MKL Based Multimodal Emotion Recognition and Sentiment Analysis

  2016cited by this paper
• Select-additive learning: Improving generalization in multimodal sentiment analysis

  2016influential reference
• Adieu features? End-to-end speech emotion recognition using a deep convolutional recurrent network

  2016cited by this paper
• Multimodal Sentiment Intensity Analysis in Videos: Facial Gestures and Verbal Messages

  2016influential reference
• Deep Unordered Composition Rivals Syntactic Methods for Text Classification

  2015influential reference
• Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks

  2015cited by this paper
• VQA: Visual Question Answering

  2015cited by this paper
• Deep Convolutional Neural Network Textual Features and Multiple Kernel Learning for Utterance-level Multimodal Sentiment Analysis

  2015influential reference
• Micro-opinion Sentiment Intensity Analysis and Summarization in Online Videos

  2015cited by this paper
• High-level feature representation using recurrent neural network for speech emotion recognition

  2015cited by this paper
• Concept-Level Sentiment Analysis with Dependency-Based Semantic Parsing: A Novel Approach

  2015cited by this paper
• Recurrent Neural Networks for Emotion Recognition in Video

  2015cited by this paper
• GloVe: Global Vectors for Word Representation

  2014cited by this paper
• COVAREP — A collaborative voice analysis repository for speech technologies

  2014influential reference
• Long-term recurrent convolutional networks for visual recognition and description

  2014cited by this paper
• Sentic patterns: Dependency-based rules for concept-level sentiment analysis

  2014cited by this paper
• Adam: A Method for Stochastic Optimization

  2014cited by this paper
• A Convolutional Neural Network for Modelling Sentences

  2014cited by this paper
• Facial Expression Recognition Using 3D Convolutional Neural Network

  2014cited by this paper
• Dependency-Based Semantic Parsing for Concept-Level Text Analysis

  2014cited by this paper
• Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank

  2013influential reference
• YouTube Movie Reviews: Sentiment Analysis in an Audio-Visual Context

  2013cited by this paper
• Wavelet Maxima Dispersion for Breathy to Tense Voice Discrimination

  2013cited by this paper
• Utterance-Level Multimodal Sentiment Analysis

  2013influential reference
• Extracting Opinion Expressions with semi-Markov Conditional Random Fields

  2012cited by this paper
• Detection of Glottal Closure Instants From Speech Signals: A Quantitative Review

  2012cited by this paper
• Lexicon-Based Methods for Sentiment Analysis

  2011cited by this paper
• Multiple Classifier Systems for the Classification of Audio-Visual Emotional States

  2011cited by this paper
• Towards multimodal sentiment analysis: harvesting opinions from the web

  2011influential reference
• Mining and summarizing customer reviews

  2004cited by this paper
• Normalized amplitude quotient for parametrization of the glottal flow.

  2002cited by this paper
• Learning to Forget: Continual Prediction with LSTM

  2000cited by this paper
• Long Short-Term Memory

  1997cited by this paper
• Parabolic spectral parameter - A new method for quantification of the glottal flow

  1997cited by this paper
• An argument for basic emotions

  1992cited by this paper
• Glottal wave analysis with Pitch Synchronous Iterative Adaptive Inverse Filtering

  1991cited by this paper
• Vocal quality factors: analysis, synthesis, and perception.

  1991cited by this paper
• Vocal intensity in speakers and singers.

  1991cited by this paper
• Proposal and evaluation of models for the glottal source waveform

  1986cited by this paper
• Facial signs of emotional experience.

  1980cited by this paper

• A Mixture-of-Experts model for multimodal emotion recognition in conversations

  2026cites this paper
• Affection-Guided Bottleneck Diffusion for Missing Modality Issue in Multimodal Affective Computing

  2026cites this paper
• StreamSense: Streaming Social Task Detection with Selective Vision-Language Model Routing

  2026cites this paper
• CrossLLM-Mamba: Multimodal State Space Fusion of LLMs for RNA Interaction Prediction

  2026cites this paper
• Robust Harmful Meme Detection under Missing Modalities via Shared Representation Learning

  2026cites this paper
• BCFNet: Bi-temporal collaborative fusion network for multi-modal humor detection

  2026cites this paper
• A Secure Federated Learning Algorithm for Emotion Recognition Towards Multimodal Speaker Signals on the Client Side

  2026cites this paper
• Affective computing–driven virtual rehabilitation: a systematic survey

  2026cites this paper
• Toward multimodal sentiment analysis with a self-supervised knowledge-augmented network

  2026cites this paper
• GCMA-Net: A Gated Cross-Modal Attention Network for Arabic Multimodal Sentiment Analysis

  2026cites this paper
• A trinity-branch parallel fusion and supervised enhancement network: A multimodal celiac disease diagnosis network based on transformer and dual-tower supervision

  2026cites this paper
• TB-MEN: Text-Centric Bidirectional Modality Enhancement Network for Multimodal Sentiment Analysis

  2026cites this paper
• Temporal-Spatial Decouple before Act: Disentangled Representation Learning for Multimodal Sentiment Analysis

  2026cites this paper
• Adaptive weighted temporal prototype network for multimodal emotion recognition

  2026cites this paper
• DCGRM-Net: Dual-Channel Guided Reconstruction Mamba Network for robust multimodal sentiment analysis

  2026cites this paper
• A Vision for Multisensory Intelligence: Sensing, Science, and Synergy

  2026cites this paper
• Feature-level interaction and adaptive fusion model based on cross-modal attention for audiovisual emotion recognition

  2026cites this paper
• CAF-Mamba: Mamba-Based Cross-Modal Adaptive Attention Fusion for Multimodal Depression Detection

  2026cites this paper
• DCER: Dual-Stage Compression and Energy-Based Reconstruction

  2026cites this paper
• SADGR: Adaptive Cross-Modal Emotion Recognition via Self-Supervised Alignment and Dynamic Gating

  2026influential citation
• Neural Representational Geometry of Feature Binding Operations

  2026cites this paper
• AG-MSA: Adaptive Gated Prompt Learning for Few-Shot Multimodal Sentiment Analysis

  2026cites this paper
• MultiModalPFN: Extending Prior-Data Fitted Networks for Multimodal Tabular Learning

  2026cites this paper
• CLCR: Cross-Level Semantic Collaborative Representation for Multimodal Learning

  2026cites this paper
• MHAMF: mamba-based emotional hyper-modal assisted multi-granularity fusion for emotion recognition in conversations

  2026cites this paper
• Graph-Prototype distillation with prototype-Guided contrastive training for multimodal emotion recognition in conversations

  2026cites this paper
• Disentangled representation learning with temporal smoothness constraints for multimodal sentiment analysis

  2026cites this paper
• Lipschitz-controlled attention fusion for stable multimodal autoencoders in industrial robotics and manufacturing

  2026cites this paper
• DynMultiDep: A Dynamic Multimodal Fusion and Multi-Scale Time Series Modeling Approach for Depression Detection

  2026cites this paper
• Meta-RoMSA: A Reciprocal Translation and Modality Balancing Framework for Robust Multimodal Sentiment Analysis Using Meta-Learning

  2026cites this paper
• Multimodal fusion in speech emotion recognition: A comprehensive review of methods and technologies

  2026cites this paper
• Step-Wise Prompting Meets Uncertainty-Aware Dynamic Fusion for Robust EEG-Visual Emotion Recognition

  2026cites this paper
• StaProDyn: A unified framework for multimodal sentiment analysis with stability-aware filtering, prompt learning enhancement, and dynamic fusion

  2026cites this paper
• Multimodal Sentiment Analysis based on Multi-channel and Symmetric Mutual Promotion Feature Fusion

  2026cites this paper
• A Unified Framework for Emotion Recognition and Sentiment Analysis via Expert-Guided Multimodal Fusion with Large Language Models

  2026cites this paper
• Integrated feature-enhanced multimodal intent detection method

  2026cites this paper
• Emotion-LLaMAv2 and MMEVerse: A New Framework and Benchmark for Multimodal Emotion Understanding

  2026influential citation
• CMPTA: Exploring the Application of Pre-Trained Large Language Models in Multimodal Sentiment Analysis

  2026cites this paper
• A Baseline Multimodal Approach to Emotion Recognition in Conversations

  2026cites this paper
• Decoupled Hierarchical Distillation for Multimodal Emotion Recognition.

  2026influential citation
• Understanding multimodal sentiment with deep modality interaction learning

  2026cites this paper
• A deep learning model for photovoltaic soiling loss prediction and estimation based on Large Kernel Cross-Attention Fusion

  2026cites this paper
• DRFusion: Enhancing balanced and sufficient multimodal learning for human emotion recognition

  2026cites this paper
• HLCN: A Hypergraph Laplace Contrastive Network for Enhanced Multimodal Sentiment Analysis

  2026cites this paper
• Tri-Subspaces Disentanglement for Multimodal Sentiment Analysis

  2026cites this paper
• SPP-SCL: Semi-Push-Pull Supervised Contrastive Learning for Image-Text Sentiment Analysis and Beyond

  2026cites this paper
• Building Prototype Evolution Pathway for Emotion Recognition in User-Generated Videos

  2026cites this paper
• RMSAGF: A Synergistic Generation and Fusion Framework for Robust Multimodal Sentiment Analysis

  2026cites this paper
• An AI-driven framework for coastal city monitoring via deep learning and earth observation

  2026cites this paper
• CaReFlow: Cyclic Adaptive Rectified Flow for Multimodal Fusion

  2026cites this paper
• Breaking the Correlation Plateau: On the Optimization and Capacity Limits of Attention-Based Regressors

  2026cites this paper
• Emotional conflict adaptation for multimodal sentiment analysis

  2026cites this paper
• A Multimodal Sentiment Analysis Approach Based on Multiview Cross-Modal Fusion

  2026influential citation
• PACC: Protocol-Aware Cross-Layer Compression for Compact Network Traffic Representation

  2026cites this paper
• AdaFuse: Adaptive Multimodal Fusion for Lung Cancer Risk Prediction via Reinforcement Learning

  2026cites this paper
• LUCID: lexicon-augmented fusion with ordinal calibration for multimodal sentiment analysis

  2026cites this paper
• FE-DHH: Fourier-enhanced dual high-order hypergraph for multimodal conversational emotion recognition

  2026cites this paper
• Grading-inspired complementary enhancing for multimodal sentiment analysis

  2026cites this paper
• GRCF: Two-Stage Groupwise Ranking and Calibration Framework for Multimodal Sentiment Analysis

  2026cites this paper
• Sentiment Analysis on Movie Reviews: A Deep Dive into Modern Techniques and Open Challenges

  2026cites this paper
• Confidence-guided dynamic sequential fusion for multimodal sentiment analysis

  2026influential citation
• Cognitive-inspired knowledge graph fusion for gradient-aligned multimodal sentiment analysis

  2026cites this paper
• A multi-scale representation and multi-level decision learning network for multimodal sentiment analysis

  2026cites this paper
• AL-HCL: Active Learning and Hierarchical Contrastive Learning for Multimodal Sentiment Analysis With Fusion Guidance

  2026cites this paper
• A Lightweight Two-Stage Attention-Gating Network for Efficient Multimodal Sentiment Analysis

  2026cites this paper
• Can Large Language Models Help Multimodal Language Analysis? MMLA: A Comprehensive Benchmark

  2025cites this paper
• Learning by Comparing: Boosting Multimodal Affective Computing through Ordinal Learning

  2025cites this paper
• Multimodal intent recognition based on text-guided cross-modal attention

  2025cites this paper
• Towards Explainable Fusion and Balanced Learning in Multimodal Sentiment Analysis

  2025cites this paper
• DeepMLF: Multimodal language model with learnable tokens for deep fusion in sentiment analysis

  2025influential citation
• HGTFM: Hierarchical Gating-Driven Transformer Fusion Model for Robust Multimodal Sentiment Analysis

  2025cites this paper
• A multimodal fusion method with interpretability for nozzle health online prediction

  2025influential citation
• RoLiVit: Feature Fusion Approach for Multimodal Sentiment Analysis Using Deep Learning

  2025cites this paper
• Cross-Aligned Fusion For Multimodal Understanding

  2025cites this paper
• CMFF_VS: A Video Summarization Extraction Model based on Cross-modal Feature Fusion

  2025cites this paper
• Multimodal sentiment analysis with text-augmented cross-modal feature interaction attention network

  2025cites this paper
• VEGAS: Towards Visually Explainable and Grounded Artificial Social Intelligence

  2025cites this paper
• Handwritten Signature Verification via Multimodal Consistency Learning

  2025cites this paper
• Emotion-Assisted multi-modal Personality Recognition using adversarial Contrastive learning

  2025cites this paper
• Bimodal Sentiment Analysis Based on a Pre-Trained Model and Masked Attention Fusion

  2025cites this paper
• TOMFuN: A tensorized optical multimodal fusion network

  2025cites this paper
• Multimodal Sentiment Analysis—A Comprehensive Survey From a Fusion Methods Perspective

  2025cites this paper
• Hierarchical Adaptive Expert for Multimodal Sentiment Analysis

  2025cites this paper
• BiMSA: Multimodal Sentiment Analysis Based on BiGRU and Bidirectional Interactive Attention

  2025cites this paper
• Multi‐view sparse attention network for glioma survival risk prediction

  2025cites this paper
• Modality-Guided Refinement Learning for Multimodal Emotion Recognition

  2025influential citation
• Multimodal sentiment analysis method based on image-text quantum transformer

  2025cites this paper
• Multimodal GRU with directed pairwise cross-modal attention for sentiment analysis

  2025cites this paper
• Leveraging CLIP Encoder for Multimodal Emotion Recognition

  2025cites this paper
• Multimodal Sentiment Analysis

  2025cites this paper
• TGFN-SD: A text-guided multimodal fusion network for swine disease diagnosis

  2025cites this paper
• TDMER: A Task-Driven Method for Multimodal Emotion Recognition

  2025influential citation
• MHSDB: A Comprehensive Benchmark for Multimodal Humor and Sarcasm Detection Leveraging Foundation Models

  2025cites this paper
• The Potential of Speech Features to Discriminate between Original and Machine-Translated Texts

  2025cites this paper
• Fine-grained Semantic Disentanglement Network for Multimodal Sarcasm Analysis

  2025cites this paper
• A disentanglement mamba network with a temporally slack reconstruction mechanism for multimodal continuous emotion recognition

  2025cites this paper
• A novel multimodal personality prediction method based on pretrained models and graph relational transformer network

  2025cites this paper
• A Novel Moisture Measurement Method for the Sintering Mixture Based on Multivariate Feature Fusion

  2025cites this paper
• Injecting Multimodal Information Into Pre-Trained Language Model for Multimodal Sentiment Analysis

  2025influential citation
• MSAmba: Exploring Multimodal Sentiment Analysis with State Space Models

  2025influential citation