Don’t count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors

Context-predicting models (more commonly known as embeddings or neural language models) are the new kids on the distributional semantics block. Despite the buzz surrounding these models, the literature is still lacking a systematic comparison of the predictive models with classic, count-vector-based distributional semantic approaches. In this paper, we perform such an extensive evaluation, on a wide range of lexical semantics tasks and across many parameter settings. The results, to our own surprise, show that the buzz is fully justified, as the context-predicting models obtain a thorough and resounding victory against their count-based counterparts.

• Publication year

  2014

• Venue

  Annual Meeting of the Association for Computational Linguistics

• Publication date

  2014-06-01

• Fields of study

  Linguistics, Computer Science

• Identifiers
  DOI 10.3115/v1/P14-1023
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Vector Space Models of Lexical Meaning

  2015cited by this paper
• Distributed Representations of Words and Phrases and their Compositionality

  2013cited by this paper
• The Expressive Power of Word Embeddings

  2013cited by this paper
• Exploiting Similarities among Languages for Machine Translation

  2013influential reference
• Multimodal Distributional Semantics

  2013cited by this paper
• Zero-Shot Learning Through Cross-Modal Transfer

  2013cited by this paper
• DeViSE: A Deep Visual-Semantic Embedding Model

  2013cited by this paper
• Efficient Estimation of Word Representations in Vector Space

  2013influential reference
• Linguistic Regularities in Continuous Space Word Representations

  2013influential reference
• Semantic Compositionality through Recursive Matrix-Vector Spaces

  2012cited by this paper
• A Comparison of Vector-based Representations for Semantic Composition

  2012cited by this paper
• Vector Space Models of Word Meaning and Phrase Meaning: A Survey

  2012cited by this paper
• Domain and Function: A Dual-Space Model of Semantic Relations and Compositions

  2012cited by this paper
• Extracting semantic representations from word co-occurrence statistics: stop-lists, stemming, and SVD

  2012cited by this paper
• Improving Word Representations via Global Context and Multiple Word Prototypes

  2012cited by this paper
• Large-scale learning of word relatedness with constraints

  2012cited by this paper
• Semantic Relatedness Using Salient Semantic Analysis

  2011cited by this paper
• Natural Language Processing (Almost) from Scratch

  2011cited by this paper
• From Frequency to Meaning: Vector Space Models of Semantics

  2010cited by this paper
• Strudel: A distributional semantic model based on properties and types

  2010cited by this paper
• Distributional Memory: A General Framework for Corpus-Based Semantics

  2010cited by this paper
• Word Representations: A Simple and General Method for Semi-Supervised Learning

  2010cited by this paper
• Unsupervised Classification with Dependency Based Word Spaces

  2009cited by this paper
• BagPack: A General Framework to Represent Semantic Relations

  2009influential reference
• Latent Dirichlet Allocation

  2009cited by this paper
• A Study on Similarity and Relatedness Using Distributional and WordNet-based Approaches

  2009cited by this paper
• A unified architecture for natural language processing: deep neural networks with multitask learning

  2008cited by this paper
• Qualia structures and their impact on the concrete noun categorization task

  2008cited by this paper
• Topics in Semantic Representation

  2007cited by this paper
• Extracting semantic representations from word co-occurrence statistics: A computational study

  2007cited by this paper
• Projected Gradient Methods for Nonnegative Matrix Factorization

  2007cited by this paper
• Dependency-Based Construction of Semantic Space Models

  2007influential reference
• The integration of syntax and semantic plausibility in a wide-coverage model of human sentence processing

  2007cited by this paper
• Topics in semantic representation.

  2007influential reference
• Attributes in lexical acquisition

  2006cited by this paper
• The Word-Space Model: using distributional analysis to represent syntagmatic and paradigmatic relations between words in high-dimensional vector spaces

  2006cited by this paper
• The Statistics of Word Cooccur-rences: Word Pairs and Collocations

  2004cited by this paper
• A Neural Probabilistic Language Model

  2003cited by this paper
• Placing search in context: the concept revisited

  2002cited by this paper
• CLUTO - A Clustering Toolkit

  2002cited by this paper
• Algorithms for Non-negative Matrix Factorization

  2000cited by this paper
• Modeling the Influence of Thematic Fit (and Other Constraints) in On-line Sentence Comprehension

  1998cited by this paper
• A Solution to Plato's Problem: The Latent Semantic Analysis Theory of Acquisition, Induction, and Representation of Knowledge.

  1997influential reference
• A Solution to Plato ' s Problem : The Latent Semantic Analysis Theory of Acquisition , Induction , and Representation of Knowledge

  1997influential reference
• Matrix computations (3rd ed.)

  1996cited by this paper
• Contextual correlates of semantic similarity

  1991cited by this paper
• Matrix computations

  1983cited by this paper
• Contextual correlates of synonymy

  1965cited by this paper

• Evaluating the impact of word embeddings on similarity scoring in practical information retrieval

  2026cites this paper
• On the scaling relationship between cloze probabilities and language model next-token prediction

  2026cites this paper
• ReFRAME or Remain: Unsupervised Lexical Semantic Change Detection with Frame Semantics

  2026cites this paper
• Constructions are influenced in their realization by the degree of activation of horizontally linked constructions: using BERT sentence embeddings to test contamination between constructions

  2026influential citation
• Measuring Social Integration Through Participation: Categorizing Organizations and Leisure Activities in the Displaced Karelians Interview Archive using LLMs

  2026cites this paper
• Transparent Semantic Change Detection with Dependency-Based Profiles

  2026cites this paper
• Multilingual prediction of semantic norms with language models: a study on English and Chinese

  2025cites this paper
• Success and failure of compositional generalisation in distributional models of language

  2025cites this paper
• Optimising window size of semantic of classification model for identification of in-text citations based on context and intent

  2025cites this paper
• Top-down pressure and bottom-up responses: a study on provincial Governments’ adoption and textual reproduction of sports industry policies in China

  2025cites this paper
• Characterizing semantic compositions in the brain: A model-driven fMRI re-analysis

  2025influential citation
• What do the differences and commonalities in doctoral dissertation acknowledgments across disciplines reveal?

  2025cites this paper
• The role of meaning in the rivalry of -ity and -ness: evidence from distributional semantics

  2025cites this paper
• Semantic Compression for Word and Sentence Embeddings using Discrete Wavelet Transform

  2025cites this paper
• Semantic enrichment of neural word embeddings: Leveraging taxonomic similarity for enhanced distributional semantics

  2025cites this paper
• Recenzió Tóth Ágoston The Company that Words Keep: Distributional Semantics című könyvéről

  2025cites this paper
• Analyzing word embeddings and their impact on semantic similarity: through extreme simulated conditions to real dataset characteristics

  2025cites this paper
• Understanding the Challenges Students Face in Non-English Programming Environments Due to the Programming Language Transition: A Case Study of Keywords in the Chinese Version of Scratch

  2025cites this paper
• Evaluating Semantic Representation Strategies for Robust Information Retrieval Matching

  2025cites this paper
• The linguistic factors of semantic transparency: Evidence from verb-to-noun derivation in French

  2025cites this paper
• Estranged Predictions: Measuring Semantic Category Disruption with Masked Language Modelling

  2025cites this paper
• Compositionality in the semantic network: a model-driven representational similarity analysis

  2025influential citation
• Advances and Challenges in Semantic Textual Similarity: A Comprehensive Survey

  2025cites this paper
• Beyond Recommendations: Intrinsic Evaluation Strategies for Item Embeddings in Recommender Systems

  2025cites this paper
• Comparing Static and Contextual Distributional Semantic Models on Intrinsic Tasks: An Evaluation on Mandarin Chinese Datasets

  2024cites this paper
• Language Models and Semantic Relations: A Dual Relationship

  2024cites this paper
• Training and evaluation of vector models for Galician

  2024influential citation
• A Fistful of Vectors: A Tool for Intrinsic Evaluation of Word Embeddings

  2024influential citation
• Visual experience modulates the sensitivity to the distributional history of words in natural language

  2024cites this paper
• Advancing language models through domain knowledge integration: a comprehensive approach to training, evaluation, and optimization of social scientific neural word embeddings

  2024cites this paper
• Investigating Idiomaticity in Word Representations

  2024cites this paper
• From Word Vectors to Multimodal Embeddings: Techniques, Applications, and Future Directions For Large Language Models

  2024cites this paper
• Large-scale cloze evaluation reveals that token prediction tasks are neither lexically nor semantically aligned

  2024cites this paper
• Revealing COVID-19's Social Dynamics: Diachronic Semantic Analysis of Vaccine and Symptom Discourse on Twitter

  2024cites this paper
• Evolution of Human–Robot Personality Similarity: A Historical Analysis from 1960 to 2019

  2024cites this paper
• BERT may help in lexicographic sense delineation

  2024cites this paper
• Patent-publication pairs for the detection of knowledge transfer from research to industry: reducing ambiguities with word embeddings and references

  2024cites this paper
• Implicit Geometry of Next-token Prediction: From Language Sparsity Patterns to Model Representations

  2024cites this paper
• By Tying Embeddings You Are Assuming the Distributional Hypothesis

  2024cites this paper
• The AI Based Design Layout for Exploring the Reader Experience through Technical Interaction for Learning Literature

  2024cites this paper
• Are LLMs Models of Distributional Semantics? A Case Study on Quantifiers

  2024cites this paper
• Distributional Semantics: Meaning Through Culture and Interaction

  2024cites this paper
• The Outcomes and Publication Standards of Research Descriptions in Document Classification: A Systematic Review

  2024cites this paper
• DID U Misbehave? A New Dataset for In-Depth Understanding of Inconspicuous Software

  2024cites this paper
• You shall know a piece by the company it keeps. Chess plays as a data for word2vec models

  2024cites this paper
• Combining Discrete Wavelet and Cosine Transforms for Efficient Sentence Embedding

  2024cites this paper
• AN OPEN DATABASE OF SENSES FOR SERBIAN POLYSEMOUS NOUNS, VERBS, AND ADJECTIVES

  2024cites this paper
• Three Studies on Predicting Word Concreteness with Embedding Vectors

  2024cites this paper
• Language in Brains, Minds, and Machines.

  2024cites this paper
• Process Knowledge Extraction and Knowledge Graph Construction Through Prompting: A Quantitative Analysis

  2024cites this paper
• Humans Learn Language from Situated Communicative Interactions. What about Machines?

  2024cites this paper
• How Well Do LLMs Identify Cultural Unity in Diversity?

  2024cites this paper
• Has Your Pretrained Model Improved? A Multi-head Posterior Based Approach

  2024cites this paper
• On the Role of Semantic Word Clusters — CluWords — in Natural Language Processing (NLP) Tasks

  2024cites this paper
• Mel2Word: A Text-Based Melody Representation for Symbolic Music Analysis

  2024cites this paper
• Lexical choice and word formation in a taboo game paradigm

  2024cites this paper
• Large language models and linguistic intentionality

  2024cites this paper
• The impact of room shape on affective states, heartrate, and creative output

  2024cites this paper
• Automatic language ability assessment method based on natural language processing

  2024cites this paper
• Exploring Word Embeddings for Text Classification: A Comparative Analysis

  2023cites this paper
• Discovering Coherent Topics from Urdu Text: A Comparative Study of Statistical Models, Clustering Techniques and Word Embedding

  2023cites this paper
• Combining unsupervised, supervised and rule-based learning: the case of detecting patient allergies in electronic health records

  2023cites this paper
• AST-MHSA : Code Summarization using Multi-Head Self-Attention

  2023cites this paper
• Family lexicon: Using language models to encode memories of personally familiar and famous people and places in the brain

  2023cites this paper
• A semi-supervised method to generate a persian dataset for suggestion classification

  2023cites this paper
• The impact of ChatGPT on human data collection: A case study involving typicality norming data

  2023cites this paper
• A discriminative lexicon approach to word comprehension, production, and processing: Maltese plurals

  2023cites this paper
• How does linguistic context influence word learning?

  2023cites this paper
• Experientially-grounded and distributional semantic vectors uncover dissociable representations of conceptual categories

  2023cites this paper
• System API Vectorization for Malware Detection

  2023cites this paper
• Distilling Semantic Concept Embeddings from Contrastively Fine-Tuned Language Models

  2023cites this paper
• A Primer on Seq2Seq Models for Generative Chatbots

  2023cites this paper
• Scene context is predictive of unconstrained object similarity judgments.

  2023cites this paper
• A Smaller and Better Word Embedding for Neural Machine Translation

  2023cites this paper
• A distributional assessment of rivalry in word formation

  2023cites this paper
• Context Synthesis Accelerates Vocabulary Learning Through Reading: The Implication of Distributional Semantic Theory on Second Language Vocabulary Research

  2023cites this paper
• More is Better: English Language Statistics are Biased Toward Addition

  2023cites this paper
• A tale of two lexica: Investigating computational pressures on word representation with neural networks

  2023cites this paper
• Static Fuzzy Bag-of-Words: a lightweight sentence embedding algorithm

  2023cites this paper
• Improving Word Embedding Using Variational Dropout

  2023cites this paper
• The challenge of representation learning: Improved accuracy in deep vision models does not come with better predictions of perceptual similarity

  2023cites this paper
• A Comprehensive Empirical Evaluation of Existing Word Embedding Approaches

  2023cites this paper
• Investigating the Extent to which Distributional Semantic Models Capture a Broad Range of Semantic Relations

  2023cites this paper
• Neural networks for learning personality traits from natural language

  2023cites this paper
• Content-based and Knowledge-enriched Representations for Classification Across Modalities: A Survey

  2023cites this paper
• Witgenstein's influence on artificial intelligence

  2023cites this paper
• Two Stage Job Title Identification System for Online Job Advertisements

  2023cites this paper
• Overlap in meaning is a stronger predictor of semantic activation in GPT-3 than in humans

  2023cites this paper
• Extraction de relations sémantiques et modèles de langue : pour une relation à double sens

  2023cites this paper
• Global Ontology Entities Embeddings

  2023cites this paper
• Using word vector models to trace conceptual change over time and space in historical

  2023cites this paper
• Spoken Word2Vec: Learning Skipgram Embeddings from Speech

  2023cites this paper
• A Study on the Relevance of Generic Word Embeddings for Sentence Classification in Hepatic Surgery

  2023cites this paper
• Research on Sentiment Analysis of Travel Reviews Based on Word2Vec and RNN

  2023cites this paper
• Exploring the role of AI in classifying, analyzing, and generating case reports on assisted suicide cases: feasibility and ethical implications

  2023cites this paper
• Two Faces of Novelty: Idea Selection in Crowdsourcing Challenges

  2023cites this paper
• Automated clustering and switching algorithms applied to semantic verbal fluency data in schizophrenia spectrum disorders

  2023influential citation
• SanskritShala: A Neural Sanskrit NLP Toolkit with Web-Based Interface for Pedagogical and Annotation Purposes

  2023cites this paper
• Embeddings for Automatic Short Answer Grading: A Scoping Review

  2023cites this paper
• Towards Turkish Word Embeddings: An Intrinsic Evaluation

  2023cites this paper