Black Holes and White Rabbits: Metaphor Identification with Visual Features

Metaphor is pervasive in our communication, which makes it an important problem for nat-ural language processing (NLP). Numerous approaches to metaphor processing have thus been proposed, all of which relied on linguistic features and textual data to construct their models. Human metaphor comprehension is, however, known to rely on both our linguistic and perceptual experience, and vision can play a particularly important role when metaphorically projecting imagery across domains. In this paper, we present the first metaphor identification method that simultaneously draws knowledge from linguistic and visual data. Our results demonstrate that it outperforms linguistic and visual models in isolation, as well as being competitive with the best-performing metaphor identification methods, that rely on hand-crafted knowledge about domains and perception.

• Publication year

  2016

• Venue

  North American Chapter of the Association for Computational Linguistics

• Publication date

  2016-06-01

• Fields of study

  Linguistics, Computer Science

• Identifiers
  DOI 10.18653/v1/N16-1020
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Metaphor as a Medium for Emotion: An Empirical Study

  2016cited by this paper
• Exploiting Image Generality for Lexical Entailment Detection

  2015cited by this paper
• Multi- and Cross-Modal Semantics Beyond Vision: Grounding in Auditory Perception

  2015cited by this paper
• Design and Evaluation of Metaphor Processing Systems

  2015cited by this paper
• Supervised Word-Level Metaphor Detection: Experiments with Concreteness and Reweighting of Examples

  2015cited by this paper
• Visual Bilingual Lexicon Induction with Transferred ConvNet Features

  2015influential reference
• Metaphor Detection with Cross-Lingual Model Transfer

  2014influential reference
• ImageNet Large Scale Visual Recognition Challenge

  2014cited by this paper
• Linguistic Regularities in Sparse and Explicit Word Representations

  2014cited by this paper
• A Quantitative Empirical Analysis of the Abstract/Concrete Distinction

  2014cited by this paper
• Caffe: Convolutional Architecture for Fast Feature Embedding

  2014cited by this paper
• Learning Image Embeddings using Convolutional Neural Networks for Improved Multi-Modal Semantics

  2014cited by this paper
• The Stanford CoreNLP Natural Language Processing Toolkit

  2014cited by this paper
• A Novel Distributional Approach to Multilingual Conceptual Metaphor Recognition

  2014cited by this paper
• Multimodal Distributional Semantics

  2013cited by this paper
• Data-Driven Metaphor Recognition and Explanation

  2013cited by this paper
• A Multimodal LDA Model integrating Textual, Cognitive and Visual Modalities

  2013cited by this paper
• Cross-Lingual Metaphor Detection Using Common Semantic Features

  2013cited by this paper
• Metaphor Identification in Large Texts Corpora

  2013influential reference
• Automatic Metaphor Detection using Large-Scale Lexical Resources and Conventional Metaphor Extraction

  2013cited by this paper
• Efficient Estimation of Word Representations in Vector Space

  2013cited by this paper
• Automatic Identification of Conceptual Metaphors With Limited Knowledge

  2013influential reference
• Metaphor Interpretation Using Paraphrases Extracted from the Web

  2013cited by this paper
• Linguistic Regularities in Continuous Space Word Representations

  2013cited by this paper
• What metaphor identification systems can tell us about metaphor-in-language

  2013influential reference
• Evaluating the Premises and Results of Four Metaphor Identification Systems

  2013influential reference
• Identifying Metaphorical Word Use with Tree Kernels

  2013influential reference
• Semantic Signatures for Example-Based Linguistic Metaphor Detection

  2013cited by this paper
• Unsupervised Metaphor Identification Using Hierarchical Graph Factorization Clustering

  2013influential reference
• Robust Extraction of Metaphor from Novel Data

  2013influential reference
• Metaphor Detection

  2013influential reference
• ImageNet classification with deep convolutional neural networks

  2012cited by this paper
• Unsupervised Metaphor Paraphrasing using a Vector Space Model

  2012cited by this paper
• Grounded Models of Semantic Representation

  2012cited by this paper
• Symbol Interdependency in Symbolic and Embodied Cognition

  2011cited by this paper
• Literal and Metaphorical Sense Identification through Concrete and Abstract Context

  2011cited by this paper
• Using Visual Information to Predict Lexical Preference

  2011cited by this paper
• Automatic Metaphor Interpretation as a Paraphrasing Task

  2010cited by this paper
• Metaphor Identification Using Verb and Noun Clustering

  2010influential reference
• Grounded cognition.

  2008cited by this paper
• A Fluid Knowledge Representation for Understanding and Generating Creative Metaphors

  2008cited by this paper
• Hunting Elusive Metaphors Using Lexical Resources.

  2007cited by this paper
• From Molecule to Metaphor - A Neural Theory of Language

  2006cited by this paper
• Catching Metaphors

  2006cited by this paper
• Learning object categories from Google's image search

  2005cited by this paper
• CorMet: A Computational, Corpus-Based Conventional Metaphor Extraction System

  2004influential reference
• Metaphors We Live by

  1982cited by this paper

• Bridging the gap, not forcing the tie: dual-space alignment and fusion framework for toxic memes detection

  2026cites this paper
• Cultural Bias Matters: A Cross-Cultural Benchmark Dataset and Sentiment-Enriched Model for Understanding Multimodal Metaphors

  2025cites this paper
• G-TBML: GAN-enhanced Transformer-based Multimodal Binding Learning Model

  2025cites this paper
• CMGBench: Benchmarking Chinese Metaphor Generation for Large Language Models

  2025cites this paper
• Modality-uncertainty-aware knowledge distillation framework for multimodal sentiment analysis

  2025cites this paper
• ImaRA: An Imaginative Frame Augmented Method for Low-Resource Multimodal Metaphor Detection and Explanation

  2025cites this paper
• PMAAN: a priority-guided multi-scale adaptive attention network for multimodal sentiment analysis

  2025cites this paper
• INJECTING PERCEPTUAL FEATURES INTO T5 FOR FIGURATIVE LANGUAGE GENERATION

  2025influential citation
• Design of a Multimodal Intelligent Public Opinion Analysis System

  2025cites this paper
• DNTB: Dual-branch network model based on transformer and Bi-LSTM for energy consumption prediction in building chiller systems

  2025cites this paper
• Class Similarity-Based Multimodal Classification under Heterogeneous Category Sets

  2025cites this paper
• SFVE: visual information enhancement metaphor detection with multimodal splitting fusion

  2025cites this paper
• CU-SEMLP: All-MLP-based multimodal interaction model for multimodal sentiment analysis

  2025cites this paper
• Multimodal metaphor recognition based on chain-of-cognition prompting

  2025cites this paper
• Multimodal Aspect-Level Sentiment Analysis Based on Attention Mechanism and Multi-Layer Perceptron

  2025cites this paper
• A diffusion-based multi-modal anomaly detection approach for data communication in TACS

  2025cites this paper
• Credibility-Aware Multi-Modal Fusion Using Probabilistic Circuits

  2024cites this paper
• What do they "meme"? A metaphor-aware multi-modal multi-task framework for fine-grained meme understanding

  2024cites this paper
• A Survey of Multi-modal Knowledge Graphs: Technologies and Trends

  2024cites this paper
• A Dataset for Metaphor Detection in Early Medieval Hebrew Poetry

  2024cites this paper
• Dynamic Evidence Decoupling for Trusted Multi-view Learning

  2024cites this paper
• Computational Modeling of Metaphor in Discourse

  2024cites this paper
• Adaptive Graph Learning for Multimodal Conversational Emotion Detection

  2024cites this paper
• The Delta of Thought: Channeling Rivers of Commonsense Knowledge in the Sea of Metaphorical Interpretations

  2024cites this paper
• CAMEL: Capturing Metaphorical Alignment with Context Disentangling for Multimodal Emotion Recognition

  2024cites this paper
• Curriculum-style Data Augmentation for LLM-based Metaphor Detection

  2024cites this paper
• A Theory Guided Scaffolding Instruction Framework for LLM-Enabled Metaphor Reasoning

  2024cites this paper
• MSCoTDet: Language-Driven Multi-Modal Fusion for Improved Multispectral Pedestrian Detection

  2024cites this paper
• Exploring Chain-of-Thought for Multi-modal Metaphor Detection

  2024cites this paper
• Can ChatGPT's Performance be Improved on Verb Metaphor Detection Tasks? Bootstrapping and Combining Tacit Knowledge

  2024influential citation
• A Novel Automatic Definition Selector and Scoring Scheme for the Identification of Filipino Metaphors

  2024cites this paper
• On the Robustness and Reliability of Late Multi-Modal Fusion using Probabilistic Circuits

  2024cites this paper
• MTFR: An universal multimodal fusion method through Modality Transfer and Fusion Refinement

  2024cites this paper
• Metaphor Detection with Context Enhancement and Curriculum Learning

  2024cites this paper
• Multi-modal Feature Fistillation Emotion Recognition Method For Social Media

  2024cites this paper
• A Survey on Integrated Sensing, Communication, and Computation

  2024cites this paper
• Recent Trends of Multimodal Affective Computing: A Survey from NLP Perspective

  2024cites this paper
• Bridging Word-Pair and Token-Level Metaphor Detection with Explainable Domain Mining

  2024cites this paper
• Jointly Modeling Inter- & Intra-Modality Dependencies for Multi-modal Learning

  2024cites this paper
• A Review of Multimodal Sentiment Analysis: Modal Fusion and Representation

  2024cites this paper
• Merely Judging Metaphor is Not Enough: Research on Reasonable Metaphor Detection

  2024cites this paper
• Generating Multimodal Metaphorical Features for Meme Understanding

  2024cites this paper
• Aspect Based Sentiment Analysis on Multimodal Data: A Transformer and Low-Rank Fusion Approach

  2024cites this paper
• Diagnosis of systemic lupus erythematosus using cross-modal specific transfer fusion technology based on infrared spectra and metabolomics.

  2024cites this paper
• Paper Bullets: Modeling Propaganda with the Help of Metaphor

  2023cites this paper
• Dispersing the clouds of doubt: can cosine similarity of word embeddings help identify relation-level metaphors in Slovene?

  2023cites this paper
• Chinese Metaphorical Relation Extraction: Dataset and Models

  2023cites this paper
• Multimodal sentiment analysis based on fusion methods: A survey

  2023cites this paper
• Multi-modal Machine Learning for Vehicle Rating Predictions Using Image, Text, and Parametric Data

  2023cites this paper
• Multi-modal deep fusion for bridge condition assessment

  2023cites this paper
• Analyzing audiovisual data for understanding user's emotion in human-computer interaction environment

  2023cites this paper
• Construction Artifacts in Metaphor Identification Datasets

  2023cites this paper
• Methods of Annotating and Identifying Metaphors in the Field of Natural Language Processing

  2023cites this paper
• Multi-modal Machine Learning in Engineering Design: A Review and Future Directions

  2023cites this paper
• Fantastic Expressions and Where to Find Them: Chinese Simile Generation with Multiple Constraints

  2023cites this paper
• Decoding Symbolism in Language Models

  2023cites this paper
• A survey on computational metaphor processing techniques: from identification, interpretation, generation to application

  2023influential citation
• MultiCMET: A Novel Chinese Benchmark for Understanding Multimodal Metaphor

  2023influential citation
• Uncertainty-Aware Multiview Deep Learning for Internet of Things Applications

  2023cites this paper
• Deep Learning-Based Knowledge Injection for Metaphor Detection: A Comprehensive Review

  2023cites this paper
• MetaMapper: Interpretable Metaphor Detection

  2023cites this paper
• Multimodal Deep Learning Crime Prediction Using Tweets

  2023cites this paper
• Perceptional and actional enrichment for metaphor detection with sensorimotor norms

  2023influential citation
• HFFD: Hybrid Fusion Based Multimodal Flood Relevance Detection

  2022cites this paper
• Identifikasi Ekspresi Idiomatik Menggunakan Distributional Semantic Based Approach dan Truth Discovery

  2022cites this paper
• Metaphorical Paraphrase Generation: Feeding Metaphorical Language Models with Literal Texts

  2022cites this paper
• DOMFN: A Divergence-Orientated Multi-Modal Fusion Network for Resume Assessment

  2022cites this paper
• Political Metaphors in U.S. Governor Speeches

  2022cites this paper
• Explainable Metaphor Identification Inspired by Conceptual Metaphor Theory

  2022influential citation
• Multimodal Fusion Remote Sensing Image–Audio Retrieval

  2022cites this paper
• Trusted Multi-View Classification With Dynamic Evidential Fusion

  2022cites this paper
• M ASKED M ETAPHOR M ODELING TO T RANSFER L ITERAL TO M ETAPHORICAL T EXT

  2022cites this paper
• Understanding, Exploiting and Improving Inter-view Relationships

  2022cites this paper
• MetaPro: A computational metaphor processing model for text pre-processing

  2022influential citation
• Word-level and phrase-level strategies for figurative text identification

  2022cites this paper
• Metaphors in Pre-Trained Language Models: Probing and Generalization Across Datasets and Languages

  2022cites this paper
• In Your Face: Sentiment Analysis of Metaphor with Facial Expressive Features

  2021cites this paper
• On the Impact of Temporal Representations on Metaphor Detection

  2021cites this paper
• Deep Graph Fusion Based Multimodal Evoked Expressions From Large-Scale Videos

  2021cites this paper
• MelBERT: Metaphor Detection via Contextualized Late Interaction using Metaphorical Identification Theories

  2021cites this paper
• Action recognition based on element-level fine-grained multi-modal fusion

  2021cites this paper
• Combining Pre-trained Word Embeddings and Linguistic Features for Sequential Metaphor Identification

  2021cites this paper
• CCMR: A Classic-enriched Connotation-aware Music Retrieval System on Social Media with Visual Inputs

  2021cites this paper
• MERMAID: Metaphor Generation with Symbolism and Discriminative Decoding

  2021cites this paper
• CTNet: Conversational Transformer Network for Emotion Recognition

  2021cites this paper
• Linguistic issues behind visual question answering

  2021cites this paper
• A Knowledge Graph Embedding Approach for Metaphor Processing

  2021cites this paper
• Probing, Generalization and Application of Metaphorical Knowledge in Pre-trained Language Models Anonymous ACL-IJCNLP submission

  2021cites this paper
• Enhancing Metaphor Detection by Gloss-based Interpretations

  2021cites this paper
• What makes the difference? An empirical comparison of fusion strategies for multimodal language analysis

  2021cites this paper
• Quantum-inspired multimodal fusion for video sentiment analysis

  2021cites this paper
• Recent advances in neural metaphor processing: A linguistic, cognitive and social perspective

  2021cites this paper
• Probing and Generalization of Metaphorical Knowledge in Pre-Trained Language Models Anonymous ACL-IJCNLP submission

  2021cites this paper
• Enhanced Metaphor Detection via Incorporation of External Knowledge Based on Linguistic Theories

  2021cites this paper
• Bimodal variational autoencoder for audiovisual speech recognition

  2021cites this paper
• Label-Enhanced Hierarchical Contextualized Representation for Sequential Metaphor Identification

  2021cites this paper
• CTFN: Hierarchical Learning for Multimodal Sentiment Analysis Using Coupled-Translation Fusion Network

  2021cites this paper
• How Metaphors Impact Political Discourse: A Large-Scale Topic-Agnostic Study Using Neural Metaphor Detection

  2021cites this paper
• Iconary: A Pictionary-Based Game for Testing Multimodal Communication with Drawings and Text

  2021cites this paper
• Verb Metaphor Detection via Contextual Relation Learning

  2021cites this paper