Model-based Word Embeddings from Decompositions of Count Matrices

This work develops a new statistical understanding of word embeddings induced from transformed count data. Using the class of hidden Markov models (HMMs) underlying Brown clustering as a generative model, we demonstrate how canonical correlation analysis (CCA) and certain count transformations permit efficient and effective recovery of model parameters with lexical semantics. We further show in experiments that these techniques empirically outperform existing spectral methods on word similarity and analogy tasks, and are also competitive with other popular methods such as WORD2VEC and GLOVE.

• Publication year

  2015

• Venue

  Annual Meeting of the Association for Computational Linguistics

• Publication date

  Unknown publication date

• Fields of study

  Mathematics, Computer Science

• Identifiers
  DOI 10.3115/v1/P15-1124
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Improving Distributional Similarity with Lessons Learned from Word Embeddings

  2015cited by this paper
• Neural Word Embedding as Implicit Matrix Factorization

  2014influential reference
• Linguistic Regularities in Sparse and Explicit Word Representations

  2014influential reference
• GloVe: Global Vectors for Word Representation

  2014cited by this paper
• A Spectral Algorithm for Learning Class-Based n-gram Models of Natural Language

  2014influential reference
• Low-Rank Tensors for Scoring Dependency Structures

  2014cited by this paper
• A Fast and Accurate Dependency Parser using Neural Networks

  2014cited by this paper
• Distributed Representations of Words and Phrases and their Compositionality

  2013cited by this paper
• Efficient Estimation of Word Representations in Vector Space

  2013influential reference
• Two Step CCA: A new spectral method for estimating vector models of words

  2012cited by this paper
• Natural Language Processing (Almost) from Scratch

  2011cited by this paper
• Latent semantic analysis

  2008cited by this paper
• A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data

  2005cited by this paper
• Canonical Correlation Analysis: An Overview with Application to Learning Methods

  2004cited by this paper
• Weighted Low-Rank Approximations

  2003cited by this paper
• Discovering word senses from text

  2002cited by this paper
• Class-Based n-gram Models of Natural Language

  1992cited by this paper
• Relation Between Poisson and Multinomial Distributions

  1953influential reference
• Theory of point estimation

  1950cited by this paper
• THE TRANSFORMATION OF POISSON, BINOMIAL AND NEGATIVE-BINOMIAL DATA

  1948cited by this paper
• Relations Between Two Sets of Variates

  1936influential reference
• IV.—On Least Squares and Linear Combination of Observations

  1936cited by this paper
• The Square Root Transformation in Analysis of Variance

  1936influential reference
• Edinburgh Research Explorer Experiments with Spectral Learning of Latent-Variable PCFGs

  year unknowncited by this paper

• Suicide Detection in Tweets Using LSTM and Transformers

  2024cites this paper
• Extrapolation of affective norms using transformer-based neural networks and its application to experimental stimuli selection

  2023cites this paper
• Social world knowledge: Modeling and applications

  2023cites this paper
• Multi-view overlapping clustering for the identification of the subject matter of legal judgments

  2023cites this paper
• Accessing Higher Dimensions for Unsupervised Word Translation

  2023cites this paper
• Sliding Window and Parallel LSTM with Attention and CNN for Sentence Alignment on Low-Resource Languages

  2021cites this paper
• Quantifying Context With and Without Statistical Language Models

  2021cites this paper
• Supervised Feature Embedding for Classification by Learning Rank-based Neighborhoods

  2021cites this paper
• Theoretical Understandings of Product Embedding for E-commerce Machine Learning

  2021cites this paper
• Score-Based Change Detection For Gradient-Based Learning Machines

  2021cites this paper
• FREDE: Linear-Space Anytime Graph Embeddings

  2020cites this paper
• Vergleichende Analyse der Word-Embedding-Verfahren Word2Vec und GloVe am Beispiel von Kundenbewertungen eines Online-Versandhändlers

  2020cites this paper
• Linear-Sample Learning of Low-Rank Distributions

  2020cites this paper
• Corrected CBOW Performs as well as Skip-gram

  2020cites this paper
• kōan: A Corrected CBOW Implementation

  2020cites this paper
• Identifying word evolution by incorporating PoS and avoiding alignment of temporal words

  2019cites this paper
• Capturing Word Semantics From Co-occurrences Using Dynamic Mutual Information

  2019cites this paper
• Word2Sense: Sparse Interpretable Word Embeddings

  2019cites this paper
• Calibrating GloVe model on the principle of Zipf's law

  2019cites this paper
• Kernel and Moment Based Prediction and Planning : Applications to Robotics and Natural Language Processing

  2018cites this paper
• Comparison of named entity recognition methodologies in biomedical documents

  2018cites this paper
• Inter-Annotator Agreement Networks

  2018cites this paper
• Probabilistic Clustering using Maximal Matrix Norm Couplings

  2018cites this paper
• Learning Word Embeddings for Low-Resource Languages by PU Learning

  2018cites this paper
• Batch IS NOT Heavy: Learning Word Representations From All Samples

  2018cites this paper
• A neural generative autoencoder for bilingual word embeddings

  2018cites this paper
• L G ] 6 F eb 2 01 8 Recovering Structured Probability Matrices

  2018cites this paper
• Mutual Information Maximization for Simple and Accurate Part-Of-Speech Induction

  2018cites this paper
• A method of inferring the relationship between Biomedical entities through correlation analysis on text

  2018cites this paper
• Evaluation and Analysis of Word Embedding Vectors of English Text Using Deep Learning Technique

  2017cites this paper
• Reconstruction of Word Embeddings from Sub-Word Parameters

  2017cites this paper
• AutoExtend: Combining Word Embeddings with Semantic Resources

  2017cites this paper
• All-but-the-Top: Simple and Effective Postprocessing for Word Representations

  2017cites this paper
• A Sub-Character Architecture for Korean Language Processing

  2017cites this paper
• Representing Sentences as Low-Rank Subspaces

  2017cites this paper
• Exploring Implicit Semantic Constraints for Bilingual Word Embeddings

  2017cites this paper
• Medical Incident Report Classification using Context-based Word Embeddings

  2017influential citation
• Explorer Encoding Prior Knowledge with Eigenword Embeddings

  2017cites this paper
• Information-Theory Interpretation of the Skip-Gram Negative-Sampling Objective Function

  2017cites this paper
• Correlation Analysis of Chronic Obstructive Pulmonary Disease (COPD) and its Biomarkers Using the Word Embeddings

  2017influential citation
• Supervised and unsupervised methods for learning representations of linguistic units

  2017cites this paper
• 3 Canonical Correlation Analysis for Deriving Word Embeddings

  2017cites this paper
• Detection of Alternative Ovarian Cancer Biomarker via Word Embedding

  2016cites this paper
• Classification Performance of Bio-Marker and Disease Word using Word Representation Models

  2016cites this paper
• Generalised Brown Clustering and Roll-Up Feature Generation

  2016cites this paper
• Detecting Optional Arguments of Verbs

  2016cites this paper
• Distributional initialization of neural networks

  2016cites this paper
• Unsupervised Part-Of-Speech Tagging with Anchor Hidden Markov Models

  2016influential citation
• Problems With Evaluation of Word Embeddings Using Word Similarity Tasks

  2016cites this paper
• Recovering Structured Probability Matrices

  2016cites this paper
• Sparse Word Embeddings Using ℓ1 Regularized Online Learning

  2016cites this paper
• Cross-Lingual Syntactic Transfer with Limited Resources

  2016cites this paper
• Canonical Correlation Analysis for Analyzing Sequences of Medical Billing Codes

  2016influential citation
• Canonical Correlation Inference for Mapping Abstract Scenes to Text

  2016cites this paper
• Nonsymbolic Text Representation

  2016cites this paper
• PSDVec: a Toolbox for Incremental and Scalable Word Embedding

  2016cites this paper
• Wore Representation Analysis of Bio-marker and Disease word

  2015influential citation
• Bilingual Distributed Word Representations from Document-Aligned Comparable Data

  2015cites this paper
• The mechanism of additive composition

  2015influential citation
• A Generative Word Embedding Model and its Low Rank Positive Semidefinite Solution

  2015cites this paper
• What’s in an Embedding? Analyzing Word Embeddings through Multilingual Evaluation

  2015cites this paper
• Encoding Prior Knowledge with Eigenword Embeddings

  2015influential citation
• Towards a Better Understanding of Predict and Count Models

  2015cites this paper
• Part-of-speech Taggers for Low-resource Languages using CCA Features

  2015cites this paper