Interpretable Semantic Vectors from a Joint Model of Brain- and Text- Based Meaning

Vector space models (VSMs) represent word meanings as points in a high dimensional space. VSMs are typically created using a large text corpora, and so represent word semantics as observed in text. We present a new algorithm (JNNSE) that can incorporate a measure of semantics not previously used to create VSMs: brain activation data recorded while people read words. The resulting model takes advantage of the complementary strengths and weaknesses of corpus and brain activation data to give a more complete representation of semantics. Evaluations show that the model 1) matches a behavioral measure of semantics more closely, 2) can be used to predict corpus data for unseen words and 3) has predictive power that generalizes across brain imaging technologies and across subjects. We believe that the model is thus a more faithful representation of mental vocabularies.

• Publication year

  2014

• Venue

  Annual Meeting of the Association for Computational Linguistics

• Publication date

  2014-06-01

• Fields of study

  Medicine, Computer Science

• Identifiers
  DOI 10.3115/v1/P14-1046PMID 26166940PMCID PMC4497373
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar, PubMed

• No claims are published for this paper.

• No concepts are published for this paper.

• The relationship between MEG and fMRI

  2014cited by this paper
• Interpretable Semantic Vectors from a Joint Model of Brain- and Text- Based Meaning

  2014cited by this paper
• Documents and Dependencies: an Exploration of Vector Space Models for Semantic Composition

  2013cited by this paper
• Inter-subject alignment of human cortical anatomy using functional connectivity

  2013cited by this paper
• Of Words, Eyes and Brains: Correlating Image-Based Distributional Semantic Models with Neural Representations of Concepts

  2013cited by this paper
• Limiting Factors for Mapping Corpus-Based Semantic Representations to Brain Activity

  2013cited by this paper
• Models of Semantic Representation with Visual Attributes

  2013cited by this paper
• Learning Effective and Interpretable Semantic Models using Non-Negative Sparse Embedding

  2012influential reference
• The fMRI BOLD signal tracks electrophysiological spectral perturbations, not event-related potentials

  2012cited by this paper
• Convex Multi-view Subspace Learning

  2012cited by this paper
• Tracking neural coding of perceptual and semantic features of concrete nouns

  2012influential reference
• What Makes Different People's Representations Alike: Neural Similarity Space Solves the Problem of Across-subject fMRI Decoding

  2012cited by this paper
• Grounded Models of Semantic Representation

  2012cited by this paper
• Selecting Corpus-Semantic Models for Neurolinguistic Decoding

  2012influential reference
• Distributional semantics from text and images

  2011cited by this paper
• Regularized nonnegative shared subspace learning

  2011cited by this paper
• From Frequency to Meaning: Vector Space Models of Semantics

  2010cited by this paper
• A Neurosemantic Theory of Concrete Noun Representation Based on the Underlying Brain Codes

  2010cited by this paper
• Factorized Latent Spaces with Structured Sparsity

  2010cited by this paper
• Reading Tea Leaves: How Humans Interpret Topic Models

  2009cited by this paper
• Removal of magnetoencephalographic artifacts with temporal signal‐space separation: Demonstration with single‐trial auditory‐evoked responses

  2009cited by this paper
• Integrating Multiple-Study Multiple-Subject fMRI Datasets Using Canonical Correlation Analysis

  2009cited by this paper
• Zero-shot Learning with Semantic Output Codes

  2009cited by this paper
• Online Learning for Matrix Factorization and Sparse Coding

  2009cited by this paper
• Matching categorical object representations in inferior temporal cortex of man and monkey.

  2008cited by this paper
• Representational Similarity Analysis – Connecting the Branches of Systems Neuroscience

  2008cited by this paper
• Predicting Human Brain Activity Associated with the Meanings of Nouns

  2008cited by this paper
• Supervised Topic Models

  2007cited by this paper
• Clinical neurophysiology of language: the MEG approach.

  2007cited by this paper
• The Word-Space Model: using distributional analysis to represent syntagmatic and paradigmatic relations between words in high-dimensional vector spaces

  2006cited by this paper
• Spatiotemporal signal space separation method for rejecting nearby interference in MEG measurements

  2006cited by this paper
• Semantic feature production norms for a large set of living and nonliving things

  2005influential reference
• The Signal Space Separation method

  2004cited by this paper
• On Discriminative vs. Generative Classifiers: A comparison of logistic regression and naive Bayes

  2001cited by this paper
• Symbol Grounding and Meaning: A Comparison of High-Dimensional and Embodied Theories of Meaning

  2000cited by this paper
• Signal-space projection method for separating MEG or EEG into components

  1997cited by this paper
• A Solution to Plato's Problem: The Latent Semantic Analysis Theory of Acquisition, Induction, and Representation of Knowledge.

  1997cited by this paper
• Word Association Norms, Mutual Information, and Lexicography

  1989cited by this paper
• Relations Between Two Sets of Variates

  1936cited by this paper

• From superposition to sparse codes: interpretable representations in neural networks

  2025cites this paper
• Identification of Neuro-semantic Representation of Concrete Nouns Using Cascaded Feature Selection

  2025cites this paper
• Brain and Cognitive Science Inspired Deep Learning: A Comprehensive Survey

  2025cites this paper
• An embedded computational framework of memory: The critical role of representations in veridical and false recall predictions

  2025cites this paper
• Aligning Brain Activity with Advanced Transformer Models: Exploring the Role of Punctuation in Semantic Processing

  2025cites this paper
• Distributed Data Solutions for Knowledge Based Systems by using Computational Linguistics and Econometric modeling

  2024cites this paper
• Language in Brains, Minds, and Machines.

  2024cites this paper
• Interpreting CLIP with Sparse Linear Concept Embeddings (SpLiCE)

  2024cites this paper
• From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI

  2022cites this paper
• Using Multitask Learning to Understand Language Processing in the Brain

  2022cites this paper
• Word Embedding Interpretation using Co-Clustering

  2022cites this paper
• Impart Contextualization to Static Word Embeddings through Semantic Relations

  2022cites this paper
• VICE: Variational Interpretable Concept Embeddings

  2022cites this paper
• Learning interpretable word embeddings via bidirectional alignment of dimensions with semantic concepts

  2022cites this paper
• Connecting Neural Response measurements & Computational Models of language: a non-comprehensive guide

  2022cites this paper
• SEMIE: SEMantically Infused Embeddings with Enhanced Interpretability for Domain-specific Small Corpus

  2021cites this paper
• Insights into Early Word Comprehension: Tracking the Neural Representations of Word Semantics in Infants

  2021cites this paper
• Narratives in economics

  2021cites this paper
• Plongements Interprétables pour la Détection de Biais Cachés (Interpretable Embeddings for Hidden Biases Detection)

  2021cites this paper
• Decoding EEG Brain Activity for Multi-Modal Natural Language Processing

  2021cites this paper
• The Fluidity of Concept Representations in Human Brain Signals

  2020cites this paper
• CogniVal in Action: An Interface for Customizable Cognitive Word Embedding Evaluation

  2020cites this paper
• Hybrid Approach for Accurate and Interpretable Representation Learning of Knowledge Graph

  2020cites this paper
• Behavioral correlates of cortical semantic representations modeled by word vectors

  2020cites this paper
• Towards Best Practices for Leveraging Human Language Processing Signals for Natural Language Processing

  2020cites this paper
• Evaluating Sparse Interpretable Word Embeddings for Biomedical Domain

  2020cites this paper
• Proceedings of the LREC 2020 Workshop on Linguistic and Neurocognitive Resources (LiNCr2020)

  2020cites this paper
• The POLAR Framework: Polar Opposites Enable Interpretability of Pre-Trained Word Embeddings

  2020cites this paper
• Data-driven models and computational tools for neurolinguistics: a language technology perspective

  2020cites this paper
• Word2Sense: Sparse Interpretable Word Embeddings

  2019cites this paper
• SECNLP: A Survey of Embeddings in Clinical Natural Language Processing

  2019cites this paper
• How the Brain Represents Language and Answers Questions? Using an AI System to Understand the Underlying Neurobiological Mechanisms

  2019cites this paper
• Robust Evaluation of Language-Brain Encoding Experiments

  2019cites this paper
• Analyzing Structures in the Semantic Vector Space: A Framework for Decomposing Word Embeddings

  2019cites this paper
• Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)

  2019cites this paper
• Blackbox Meets Blackbox: Representational Similarity & Stability Analysis of Neural Language Models and Brains

  2019cites this paper
• On the Semantic Interpretability of Artificial Intelligence Models

  2019cites this paper
• Inducing brain-relevant bias in natural language processing models

  2019cites this paper
• Event Structure In Vision And Language

  2019cites this paper
• An Integrated Neural Decoder of Linguistic and Experiential Meaning

  2019cites this paper
• Word Embedding Visualization Via Dictionary Learning

  2019cites this paper
• Transformation of Dense and Sparse Text Representations

  2019cites this paper
• Measuring and improving interpretability of word embeddings using lexical resources

  2019influential citation
• Studying language in context using the temporal generalization method

  2019cites this paper
• Arabic Named Entity Recognition using Word Representations

  2018cites this paper
• Imparting interpretability to word embeddings while preserving semantic structure

  2018cites this paper
• Using Sparse Semantic Embeddings Learned from Multimodal Text and Image Data to Model Human Conceptual Knowledge

  2018influential citation
• Neural Activation Semantic Models: Computational lexical semantic models of localized neural activations

  2018cites this paper
• A computational system estimating perception evoked by arbitrary visual inputs on the basis of modeling the perceptual representation in the brain

  2018cites this paper
• Semantic attributes are encoded in human electrocorticographic signals during visual object recognition

  2017cites this paper
• Commentary on Kemmerer: the challenges and rewards of trying to combine linguistics and cognitive neuroscience

  2017cites this paper
• Visually Grounded and Textual Semantic Models Differentially Decode Brain Activity Associated with Concrete and Abstract Nouns

  2017cites this paper
• Decoding the neural representation of story meanings across languages

  2017cites this paper
• Semantic Structure and Interpretability of Word Embeddings

  2017cites this paper
• Predicting Word from Brain Activity Using Joint Sparse Embedding with Domain Adaptation

  2017cites this paper
• SPINE: SParse Interpretable Neural Embeddings

  2017cites this paper
• Rotated Word Vector Representations and their Interpretability

  2017cites this paper
• Evaluation method of word embedding by roots and affixes

  2016cites this paper
• Exploring Semantic Representation in Brain Activity Using Word Embeddings

  2016cites this paper
• Sparsifying Word Representations for Deep Unordered Sentence Modeling

  2016cites this paper
• A comparative evaluation of off-the-shelf distributed semantic representations for modelling behavioural data

  2016cites this paper
• Sparse Models of Natural Language Text

  2015cites this paper
• The Time and Location of Natural Reading Processes in the Brain

  2015cites this paper
• A Hybrid Distributional and Knowledge-based Model of Lexical Semantics

  2015cites this paper
• Comparing Hebbian Semantic Vectors Across Language

  2015cites this paper
• Proposal : Beyond the Distributional Hypothesis

  2015influential citation
• Projecting attitudes ∗

  2015cites this paper
• Évaluation sur mesure de modèles distributionnels sur un corpus spécialisé : comparaison des approches par contextes syntaxiques et par fenêtres graphiques [Tailor-made evaluation of distributional models on a specialized corpus: comparison of syntactic context and graphical window approaches]

  2015cites this paper
• Non-distributional Word Vector Representations

  2015cites this paper
• Combining Language and Vision with a Multimodal Skip-gram Model

  2015cites this paper
• Corpora and Cognition: The Semantic Composition of Adjectives and Nouns in the Human Brain

  2015cites this paper
• Sparse Overcomplete Word Vector Representations

  2015cites this paper
• Information and Incrementality in Syntactic Bootstrapping

  2015cites this paper
• Learning Word Representations with Hierarchical Sparse Coding

  2014cites this paper
• Interpretable Semantic Vectors from a Joint Model of Brain- and Text- Based Meaning

  2014cites this paper
• Advances in spectral learning with applications to text analysis and brain imaging

  2014cites this paper