Bilingual Learning of Multi-sense Embeddings with Discrete Autoencoders

We present an approach to learning multi-sense word embeddings relying both on monolingual and bilingual information. Our model consists of an encoder, which uses monolingual and bilingual context (i.e. a parallel sentence) to choose a sense for a given word, and a decoder which predicts context words based on the chosen sense. The two components are estimated jointly. We observe that the word representations induced from bilingual data outperform the monolingual counterparts across a range of evaluation tasks, even though crosslingual information is not available at test time.

• Publication year

  2016

• Venue

  North American Chapter of the Association for Computational Linguistics

• Publication date

  2016-03-30

• Fields of study

  Mathematics, Linguistics, Computer Science

• Identifiers
  DOI 10.18653/v1/N16-1160arXiv 1603.09128
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Discrete-State Variational Autoencoders for Joint Discovery and Factorization of Relations

  2016cited by this paper
• Retrofitting Sense-Specific Word Vectors Using Parallel Text

  2016cited by this paper
• Improving unsupervised vector-space thematic fit evaluation via role-filler prototype clustering

  2015cited by this paper
• Breaking Sticks and Ambiguities with Adaptive Skip-gram

  2015cited by this paper
• Learning to Represent Words in Context with Multilingual Supervision

  2015cited by this paper
• Evaluation of Word Vector Representations by Subspace Alignment

  2015cited by this paper
• Deep Multilingual Correlation for Improved Word Embeddings

  2015cited by this paper
• Infinite Dimensional Word Embeddings

  2015cited by this paper
• Do Multi-Sense Embeddings Improve Natural Language Understanding?

  2015influential reference
• Linguistic Regularities in Sparse and Explicit Word Representations

  2014cited by this paper
• Don’t count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors

  2014cited by this paper
• An Autoencoder Approach to Learning Bilingual Word Representations

  2014cited by this paper
• A Unified Model for Word Sense Representation and Disambiguation

  2014cited by this paper
• An Unsupervised Model for Instance Level Subcategorization Acquisition

  2014cited by this paper
• SimLex-999: Evaluating Semantic Models With (Genuine) Similarity Estimation

  2014cited by this paper
• Improving Vector Space Word Representations Using Multilingual Correlation

  2014cited by this paper
• Conditional Random Field Autoencoders for Unsupervised Structured Prediction

  2014cited by this paper
• Community Evaluation and Exchange of Word Vectors at wordvectors.org

  2014cited by this paper
• BilBOWA: Fast Bilingual Distributed Representations without Word Alignments

  2014cited by this paper
• Fast and Robust Neural Network Joint Models for Statistical Machine Translation

  2014cited by this paper
• Tailoring Continuous Word Representations for Dependency Parsing

  2014cited by this paper
• Lexicon Infused Phrase Embeddings for Named Entity Resolution

  2014cited by this paper
• A Probabilistic Model for Learning Multi-Prototype Word Embeddings

  2014cited by this paper
• Leveraging Monolingual Data for Crosslingual Compositional Word Representations

  2014cited by this paper
• Unsupervised Induction of Semantic Roles within a Reconstruction-Error Minimization Framework

  2014cited by this paper
• Embedding Word Similarity with Neural Machine Translation

  2014cited by this paper
• Efficient Non-parametric Estimation of Multiple Embeddings per Word in Vector Space

  2014influential reference
• Learning Sense-specific Word Embeddings By Exploiting Bilingual Resources

  2014cited by this paper
• Multilingual Models for Compositional Distributed Semantics

  2014cited by this paper
• Bilingually-constrained Phrase Embeddings for Machine Translation

  2014cited by this paper
• Distributed Representations of Words and Phrases and their Compositionality

  2013influential reference
• An Information Theoretic Approach to Bilingual Word Clustering

  2013cited by this paper
• Better Word Representations with Recursive Neural Networks for Morphology

  2013cited by this paper
• Efficient Estimation of Word Representations in Vector Space

  2013influential reference
• Bilingual Word Embeddings for Phrase-Based Machine Translation

  2013cited by this paper
• Findings of the 2013 Workshop on Statistical Machine Translation

  2013influential reference
• The Joy of Parallelism with CzEng 1.0

  2012influential reference
• Representation Learning: A Review and New Perspectives

  2012cited by this paper
• Crosslingual Induction of Semantic Roles

  2012cited by this paper
• Cross-lingual Word Clusters for Direct Transfer of Linguistic Structure

  2012cited by this paper
• Inducing Crosslingual Distributed Representations of Words

  2012cited by this paper
• Distributional Semantics in Technicolor

  2012cited by this paper
• Large-scale learning of word relatedness with constraints

  2012cited by this paper
• The CMU-Avenue French-English Translation System

  2012cited by this paper
• Unsupervised Translation Sense Clustering

  2012cited by this paper
• Improving Word Representations via Global Context and Multiple Word Prototypes

  2012influential reference
• Natural Language Processing (Almost) from Scratch

  2011cited by this paper
• A word at a time: computing word relatedness using temporal semantic analysis

  2011cited by this paper
• Adaptive Subgradient Methods for Online Learning and Stochastic Optimization

  2011influential reference
• Word Representations: A Simple and General Method for Semi-Supervised Learning

  2010cited by this paper
• Posterior Regularization for Structured Latent Variable Models

  2010cited by this paper
• Climbing the Tower of Babel: Unsupervised Multilingual Learning

  2010cited by this paper
• Multi-Prototype Vector-Space Models of Word Meaning

  2010cited by this paper
• cdec: A Decoder, Alignment, and Learning Framework for Finite- State and Context-Free Translation Models

  2010cited by this paper
• Multilingual Part-of-Speech Tagging: Two Unsupervised Approaches

  2009cited by this paper
• The Oxford Guide to Practical Lexicography

  2009cited by this paper
• Phrase Clustering for Discriminative Learning

  2009cited by this paper
• Findings of the 2009 Workshop on Statistical Machine Translation

  2009influential reference
• A Study on Similarity and Relatedness Using Distributional and WordNet-based Approaches

  2009cited by this paper
• Toward using confidence intervals to compare correlations.

  2007cited by this paper
• Verb similarity on the taxonomy of WordNet

  2006cited by this paper
• Europarl: A Parallel Corpus for Statistical Machine Translation

  2005cited by this paper
• Word Sense Disambiguation: The State of the Art

  2005cited by this paper
• Word Sense Acquisition from Bilingual Comparable Corpora

  2003cited by this paper
• Exploiting Parallel Texts for Word Sense Disambiguation: An Empirical Study

  2003cited by this paper
• An Unsupervised Method for Word Sense Tagging using Parallel Corpora

  2002cited by this paper
• Placing search in context: the concept revisited

  2002cited by this paper
• Word Sense Disambiguation Using a Second Language Monolingual Corpus

  1994cited by this paper
• Word-Sense Disambiguation Using Statistical Methods

  1991cited by this paper
• Contextual correlates of semantic similarity

  1991cited by this paper
• Learning internal representations by error propagation

  1986cited by this paper
• Contextual correlates of synonymy

  1965cited by this paper
• © 1999 Kluwer Academic Publishers. Printed in the Netherlands Cross-lingual Sense Determination: Can It Work?

  year unknowncited by this paper

• From Word Types to Tokens and Back: A Survey of Approaches to Word Meaning Representation and Interpretation

  2022cites this paper
• Weakly Supervised Text Classification using Supervision Signals from a Language Model

  2022cites this paper
• Towards Multi-Sense Cross-Lingual Alignment of Contextual Embeddings

  2021cites this paper
• Embeddings in Natural Language Processing: Theory and Advances in Vector Representations of Meaning

  2020cites this paper
• Decomposing Word Embedding with the Capsule Network

  2020cites this paper
• Which Evaluations Uncover Sense Representations that Actually Make Sense?

  2020influential citation
• A Variational Approach to Unsupervised Sentiment Analysis

  2020cites this paper
• Debiasing Gender biased Hindi Words with Word-embedding

  2019cites this paper
• Exploiting Cross-Lingual Representations For Natural Language Processing

  2019cites this paper
• A Variational Approach to Weakly Supervised Document-Level Multi-Aspect Sentiment Classification

  2019cites this paper
• On the Importance of Distinguishing Word Meaning Representations: A Case Study on Reverse Dictionary Mapping

  2019cites this paper
• Deep Generative Model for Joint Alignment and Word Representation

  2018cites this paper
• Inducing and Embedding Senses with Scaled Gumbel Softmax

  2018cites this paper
• From Word to Sense Embeddings: A Survey on Vector Representations of Meaning

  2018cites this paper
• Using a Chinese Lexicon to Learn Sense Embeddings and Measure Semantic Similarity

  2018cites this paper
• CLUSE: Cross-Lingual Unsupervised Sense Embeddings

  2018influential citation
• Spot the Odd Man Out: Exploring the Associative Power of Lexical Resources

  2018cites this paper
• AspEm: Embedding Learning by Aspects in Heterogeneous Information Networks

  2018cites this paper
• Harnessing sense-level information for semantically augmented knowledge extraction

  2018cites this paper
• Learning Gender-Neutral Word Embeddings

  2018cites this paper
• Inducing and Embedding Senses with Scaled Gumbel Softmax

  2018influential citation
• Combining Explicit and Implicit Semantic Similarity Information for Word Embeddings

  2018cites this paper
• You Shall Know the Most Frequent Sense by the Company it Keeps

  2018cites this paper
• Towards a Seamless Integration of Word Senses into Downstream NLP Applications

  2017cites this paper
• EuroSense: Automatic Harvesting of Multilingual Sense Annotations from Parallel Text

  2017cites this paper
• Handling Homographs in Neural Machine Translation

  2017cites this paper
• MUSE: Modularizing Unsupervised Sense Embeddings

  2017cites this paper
• Beyond Bilingual: Multi-sense Word Embeddings using Multilingual Context

  2017influential citation
• Learning to Embed Words in Context for Syntactic Tasks

  2017cites this paper
• A Simple Approach to Learn Polysemous Word Embeddings

  2017cites this paper
• A Variational Autoencoding Approach for Inducing Cross-lingual Word Embeddings

  2017cites this paper
• IMPORTANT NOTE: You are advised to consult the publisher's version (publisher's PDF) if you wish to cite from it. Please check the document version below.

  2017cites this paper
• One Model to Rule them all: Multitask and Multilingual Modelling for Lexical Analysis

  2017cites this paper
• Interopérabilité sémantique multilingue des ressources lexicales en données lexicales liées ouvertes. (Semantic Interoperability of Multilingual Lexical Resources as Lexical Linked Data)

  2016cites this paper
• Semantic Representations of Word Senses and Concepts

  2016cites this paper
• Multi-phase Word Sense Embedding Learning Using a Corpus and a Lexical Ontology

  2016cites this paper
• Semi Supervised Preposition-Sense Disambiguation using Multilingual Data

  2016cites this paper
• Graph-Based Bilingual Word Embedding for Statistical Machine Translation

  2016cites this paper
• Language classification from bilingual word embedding graphs

  2016cites this paper
• Learning Word Sense Embeddings from Word Sense Definitions

  2016cites this paper
• HyperLex: A Large-Scale Evaluation of Graded Lexical Entailment

  2016cites this paper
• Semantic Parsing with Semi-Supervised Sequential Autoencoders

  2016cites this paper
• Embedding Words and Senses Together via Joint Knowledge-Enhanced Training

  2016cites this paper
• Empirical studies on word representations

  2016cites this paper