Learning to Embed Words in Context for Syntactic Tasks

We present models for embedding words in the context of surrounding words. Such models, which we refer to as token embeddings, represent the characteristics of a word that are specific to a given context, such as word sense, syntactic category, and semantic role. We explore simple, efficient token embedding models based on standard neural network architectures. We learn token embeddings on a large amount of unannotated text and evaluate them as features for part-of-speech taggers and dependency parsers trained on much smaller amounts of annotated data. We find that predictors endowed with token embeddings consistently outperform baseline predictors across a range of context window and training set sizes.

• Publication year

  2017

• Venue

  Rep4NLP@ACL

• Publication date

  2017-06-01

• Fields of study

  Linguistics, Computer Science

• Identifiers
  DOI 10.18653/v1/W17-2632arXiv 1706.02807
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Semi-supervised sequence tagging with bidirectional language models

  2017cited by this paper
• Semi Supervised Preposition-Sense Disambiguation using Multilingual Data

  2016cited by this paper
• Context-dependent word representation for neural machine translation

  2016cited by this paper
• Bilingual Learning of Multi-sense Embeddings with Discrete Autoencoders

  2016cited by this paper
• K-Embeddings: Learning Conceptual Embeddings for Words using Context

  2016cited by this paper
• Theano: A Python framework for fast computation of mathematical expressions

  2016cited by this paper
• context2vec: Learning Generic Context Embedding with Bidirectional LSTM

  2016cited by this paper
• Sense-Aaware Semantic Analysis: A Multi-Prototype Word Representation Model Using Wikipedia

  2015cited by this paper
• A Hierarchical Neural Autoencoder for Paragraphs and Documents

  2015cited by this paper
• Topical Word Embeddings

  2015cited by this paper
• Learning to Represent Words in Context with Multilingual Supervision

  2015cited by this paper
• Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation

  2015cited by this paper
• Do Multi-Sense Embeddings Improve Natural Language Understanding?

  2015cited by this paper
• Neural context embeddings for automatic discovery of word senses

  2015cited by this paper
• Semi-supervised Sequence Learning

  2015cited by this paper
• Lasagne: First release.

  2015cited by this paper
• Ontologically Grounded Multi-sense Representation Learning for Semantic Vector Space Models

  2015cited by this paper
• Learning Word Representation Considering Proximity and Ambiguity

  2014cited by this paper
• A Simple and Efficient Method to Generate Word Sense Representations

  2014cited by this paper
• GloVe: Global Vectors for Word Representation

  2014cited by this paper
• A Unified Model for Word Sense Representation and Disambiguation

  2014cited by this paper
• A Dependency Parser for Tweets

  2014influential reference
• Tailoring Continuous Word Representations for Dependency Parsing

  2014cited by this paper
• Sequence to Sequence Learning with Neural Networks

  2014cited by this paper
• A Probabilistic Model for Learning Multi-Prototype Word Embeddings

  2014cited by this paper
• Efficient Non-parametric Estimation of Multiple Embeddings per Word in Vector Space

  2014cited by this paper
• Learning Sense-specific Word Embeddings By Exploiting Bilingual Resources

  2014cited by this paper
• Speech recognition with deep recurrent neural networks

  2013cited by this paper
• Improved Part-of-Speech Tagging for Online Conversational Text with Word Clusters

  2013influential reference
• Bilingual Word Embeddings for Phrase-Based Machine Translation

  2013cited by this paper
• Distributed Representations of Words and Phrases and their Compositionality

  2013cited by this paper
• Improving Word Representations via Global Context and Multiple Word Prototypes

  2012cited by this paper
• LSTM Neural Networks for Language Modeling

  2012cited by this paper
• Natural Language Processing (Almost) from Scratch

  2011influential reference
• Text Normalization in Social Media: Progress, Problems and Applications for a Pre-Processing System of Casual English

  2011cited by this paper
• Word Representations: A Simple and General Method for Semi-Supervised Learning

  2010cited by this paper
• Part-of-Speech Tagging for Twitter: Annotation, Features, and Experiments

  2010cited by this paper
• Multi-Prototype Vector-Space Models of Word Meaning

  2010cited by this paper
• Visualizing Data using t-SNE

  2008cited by this paper
• Learning to Forget: Continual Prediction with LSTM

  2000cited by this paper
• Long Short-Term Memory

  1997cited by this paper
• Building a Large Annotated Corpus of English: The Penn Treebank

  1993cited by this paper

• Learning Energy-Based Approximate Inference Networks for Structured Applications in NLP

  2021cites this paper
• The Paradigm Discovery Problem

  2020cites this paper
• Towards syntax-aware token embeddings

  2020influential citation
• An Exploration of Arbitrary-Order Sequence Labeling via Energy-Based Inference Networks

  2020cites this paper
• Benchmarking Approximate Inference Methods for Neural Structured Prediction

  2019cites this paper
• Implicit Discourse Relation Classification with Syntax-Aware Contextualized Word Representations

  2019cites this paper
• Memorizing All for Implicit Discourse Relation Recognition

  2019cites this paper
• Improving Joint Training of Inference Networks and Structured Prediction Energy Networks

  2019cites this paper
• From lexical towards contextualized meaning representation

  2019cites this paper
• Phonetic-and-Semantic Embedding of Spoken words with Applications in Spoken Content Retrieval

  2018influential citation
• Exploring the effects of diacritization on Arabic frequency counts

  2018cites this paper
• Noise-Robust Morphological Disambiguation for Dialectal Arabic

  2018cites this paper
• Learning Approximate Inference Networks for Structured Prediction

  2018cites this paper
• On Learning Better Embeddings from Chinese Clinical Records: Study on Combining In-Domain and Out-Domain Data

  2018cites this paper
• Addressing Noise in Multidialectal Word Embeddings

  2018cites this paper
• Complementary Strategies for Low Resourced Morphological Modeling

  2018cites this paper
• Deep Generative Model for Joint Alignment and Word Representation

  2018cites this paper