Greedy Transition-Based Dependency Parsing with Stack LSTMs

We introduce a greedy transition-based parser that learns to represent parser states using recurrent neural networks. Our primary innovation that enables us to do this efficiently is a new control structure for sequential neural networks—the stack long short-term memory unit (LSTM). Like the conventional stack data structures used in transition-based parsers, elements can be pushed to or popped from the top of the stack in constant time, but, in addition, an LSTM maintains a continuous space embedding of the stack contents. Our model captures three facets of the parser’s state: (i) unbounded look-ahead into the buffer of incoming words, (ii) the complete history of transition actions taken by the parser, and (iii) the complete contents of the stack of partially built tree fragments, including their internal structures. In addition, we compare two different word representations: (i) standard word vectors based on look-up tables and (ii) character-based models of words. Although standard word embedding models work well in all languages, the character-based models improve the handling of out-of-vocabulary words, particularly in morphologically rich languages. Finally, we discuss the use of dynamic oracles in training the parser. During training, dynamic oracles alternate between sampling parser states from the training data and from the model as it is being learned, making the model more robust to the kinds of errors that will be made at test time. Training our model with dynamic oracles yields a linear-time greedy parser with very competitive performance.

• Publication year

  2017

• Venue

  International Conference on Computational Logic

• Publication date

  2017-06-01

• Fields of study

  Computer Science

• Identifiers
  DOI 10.1162/COLI_a_00285
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Greedy, Joint Syntactic-Semantic Parsing with Stack LSTMs

  2016cited by this paper
• Many Languages, One Parser

  2016cited by this paper
• Recurrent Neural Network Grammars

  2016cited by this paper
• Globally Normalized Transition-Based Neural Networks

  2016influential reference
• Efficient Structured Inference for Transition-Based Parsing with Neural Networks and Error States

  2016influential reference
• Stack-propagation: Improved Representation Learning for Syntax

  2016cited by this paper
• Neural Architectures for Named Entity Recognition

  2016cited by this paper
• Ask Me Anything: Dynamic Memory Networks for Natural Language Processing

  2015cited by this paper
• Reinforcement Learning Neural Turing Machines

  2015cited by this paper
• Improved Transition-based Parsing by Modeling Characters instead of Words with LSTMs

  2015cited by this paper
• Transition-Based Dependency Parsing with Stack Long Short-Term Memory

  2015influential reference
• Learning to Transduce with Unbounded Memory

  2015influential reference
• Transition-based Dependency DAG Parsing Using Dynamic Oracles

  2015cited by this paper
• Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks

  2015influential reference
• Joint Learning of Character and Word Embeddings

  2015cited by this paper
• Structured Training for Neural Network Transition-Based Parsing

  2015cited by this paper
• An Efficient Dynamic Oracle for Unrestricted Non-Projective Parsing

  2015cited by this paper
• Universal Dependencies 1.4

  2015cited by this paper
• Two/Too Simple Adaptations of Word2Vec for Syntax Problems

  2015influential reference
• LSTM: A Search Space Odyssey

  2015cited by this paper
• Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation

  2015cited by this paper
• A Neural Probabilistic Structured-Prediction Model for Transition-Based Dependency Parsing

  2015cited by this paper
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks

  2015cited by this paper
• Non-Deterministic Oracles for Unrestricted Non-Projective Transition-Based Dependency Parsing

  2015cited by this paper
• Bayesian Optimization of Text Representations

  2015cited by this paper
• Boosting Named Entity Recognition with Neural Character Embeddings

  2015cited by this paper
• Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets

  2015cited by this paper
• Incremental Recurrent Neural Network Dependency Parser with Search-based Discriminative Training

  2015influential reference
• Transition-based Neural Constituent Parsing

  2015cited by this paper
• Character-Aware Neural Language Models

  2015cited by this paper
• Compositional Morphology for Word Representations and Language Modelling

  2014cited by this paper
• Tailoring Continuous Word Representations for Dependency Parsing

  2014cited by this paper
• Global Belief Recursive Neural Networks

  2014cited by this paper
• Feature Embedding for Dependency Parsing

  2014cited by this paper
• Efficient Transfer Learning Method for Automatic Hyperparameter Tuning

  2014cited by this paper
• The Inside-Outside Recursive Neural Network model for Dependency Parsing

  2014cited by this paper
• MaltOptimizer: Fast and effective parser optimization

  2014cited by this paper
• Introducing the IMS-Wrocław-Szeged-CIS entry at the SPMRL 2014 Shared Task: Reranking and Morpho-syntax meet Unlabeled Data

  2014influential reference
• Neural Turing Machines

  2014cited by this paper
• Grammar as a Foreign Language

  2014cited by this paper
• A Polynomial-Time Dynamic Oracle for Non-Projective Dependency Parsing

  2014cited by this paper
• New Directions in Vector Space Models of Meaning

  2014cited by this paper
• A Fast and Accurate Dependency Parser using Neural Networks

  2014cited by this paper
• Introducing the SPMRL 2014 Shared Task on Parsing Morphologically-rich Languages

  2014cited by this paper
• Joint Incremental Disfluency Detection and Dependency Parsing

  2014cited by this paper
• Automatic Feature Selection for Agenda-Based Dependency Parsing

  2014cited by this paper
• Grounded Compositional Semantics for Finding and Describing Images with Sentences

  2014cited by this paper
• Normalizing tweets with edit scripts and recurrent neural embeddings

  2014cited by this paper
• A Tabular Method for Dynamic Oracles in Transition-Based Parsing

  2014cited by this paper
• Learning Character-level Representations for Part-of-Speech Tagging

  2014cited by this paper
• Sequence to Sequence Learning with Neural Networks

  2014cited by this paper
• Memory Networks

  2014cited by this paper
• Squibs: Going to the Roots of Dependency Parsing

  2013cited by this paper
• Transition-based Dependency Parsing with Selectional Branching

  2013cited by this paper
• Training Deterministic Parsers with Non-Deterministic Oracles

  2013influential reference
• Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank

  2013cited by this paper
• Effective Morphological Feature Selection with MaltOptimizer at the SPMRL 2013 Shared Task

  2013influential reference
• Generating Sequences With Recurrent Neural Networks

  2013cited by this paper
• How to Construct Deep Recurrent Neural Networks

  2013cited by this paper
• Efficient Implementation of Beam-Search Incremental Parsers

  2013cited by this paper
• Joint Morphological and Syntactic Analysis for Richly Inflected Languages

  2013cited by this paper
• Overview of the SPMRL 2013 Shared Task: A Cross-Framework Evaluation of Parsing Morphologically Rich Languages

  2013influential reference
• Efficient Higher-Order CRFs for Morphological Tagging

  2013cited by this paper
• A Non-Monotonic Arc-Eager Transition System for Dependency Parsing

  2013cited by this paper
• Parsing with Compositional Vector Grammars

  2013cited by this paper
• (Re)ranking Meets Morphosyntax: State-of-the-art Results from the SPMRL 2013 Shared Task

  2013influential reference
• Distributed Representations of Words and Phrases and their Compositionality

  2013cited by this paper
• The Role of Syntax in Vector Space Models of Compositional Semantics

  2013cited by this paper
• Transition-based Dependency Parsing Using Recursive Neural Networks

  2013cited by this paper
• Preparing Korean Data for the Shared Task on Parsing Morphologically Rich Languages

  2013cited by this paper
• Dynamic-oracle Transition-based Parsing with Calibrated Probabilistic Output

  2013cited by this paper
• A Dynamic Oracle for Arc-Eager Dependency Parsing

  2012influential reference
• An investigation of imitation learning algorithms for structured prediction

  2012cited by this paper
• Imitation Learning by Coaching

  2012cited by this paper
• A Transition-Based System for Joint Part-of-Speech Tagging and Labeled Non-Projective Dependency Parsing

  2012cited by this paper
• Making Ellipses Explicit in Dependency Conversion for a German Treebank

  2012cited by this paper
• Deep Sparse Rectifier Neural Networks

  2011cited by this paper
• Joint Hebrew Segmentation and Parsing using a PCFGLA Lattice Parser

  2011cited by this paper
• Transition-based Dependency Parsing with Rich Non-local Features

  2011cited by this paper
• Dynamic Pooling and Unfolding Recursive Autoencoders for Paraphrase Detection

  2011cited by this paper
• Dynamic Programming Algorithms for Transition-Based Dependency Parsers

  2011cited by this paper
• Towards a Bank of Constituent Parse Trees for Polish

  2010cited by this paper
• A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning

  2010cited by this paper
• Understanding the difficulty of training deep feedforward neural networks

  2010cited by this paper
• Recurrent neural network based language model

  2010cited by this paper
• Hungarian Dependency Treebank

  2010cited by this paper
• Non-Projective Dependency Parsing in Expected Linear Time

  2009cited by this paper
• Search-based structured prediction

  2009cited by this paper
• The CoNLL-2009 Shared Task: Syntactic and Semantic Dependencies in Multiple Languages

  2009cited by this paper
• Algorithms for Deterministic Incremental Dependency Parsing

  2008cited by this paper
• A Single Generative Model for Joint Morphological Segmentation and Syntactic Parsing

  2008cited by this paper
• A Tale of Two Parsers: Investigating and Combining Graph-based and Transition-based Dependency Parsing

  2008cited by this paper
• Three new graphical models for statistical language modelling

  2007cited by this paper
• A Latent Variable Model for Generative Dependency Parsing

  2007influential reference
• Incremental Non-Projective Dependency Parsing

  2007cited by this paper
• Constituent Parsing with Incremental Sigmoid Belief Networks

  2007cited by this paper
• Joint Morphological and Syntactic Disambiguation

  2007cited by this paper
• Multi-dimensional Recurrent Neural Networks

  2007cited by this paper
• Labeled Pseudo-Projective Dependency Parsing with Support Vector Machines

  2006influential reference
• Integrated Morphological and Syntactic Disambiguation for Modern Hebrew

  2006cited by this paper

• Hybrid graph structure learning for improving semantic dependency parsing with robust graph neural networks

  2025cites this paper
• Deep Learning for Time Series Forecasting: Review and Applications in Geotechnics and Geosciences

  2025cites this paper
• Hybrid embeddings for transition-based dependency parsing of free word order languages

  2023cites this paper
• Predicting the number of customer transactions using stacked LSTM recurrent neural networks

  2021cites this paper
• A survey of syntactic-semantic parsing based on constituent and dependency structures

  2020cites this paper
• Transition-based DRS Parsing Using Stack-LSTMs

  2019cites this paper
• Parallelizable Stack Long Short-Term Memory

  2019cites this paper
• Explore a deep learning multi-output neural network for regional multi-step-ahead air quality forecasts

  2019cites this paper
• Domain Information Enhanced Dependency Parser

  2019cites this paper
• Proceedings of the IWCS Shared Task on Semantic Parsing

  2019cites this paper
• Analyse automatique par transitions pour l'identification des expressions polylexicales. (Automatic transition-based analysis for multiword expression identification)

  2019cites this paper
• LSTM Easy-first Dependency Parsing with Pre-trained Word Embeddings and Character-level Word Embeddings in Vietnamese

  2018cites this paper
• IBM Research at the CoNLL 2018 Shared Task on Multilingual Parsing

  2018cites this paper
• Arc-Standard Spinal Parsing with Stack-LSTMs

  2017cites this paper
• Effective Online Reordering with Arc-Eager Transitions

  2017cites this paper
• KRISTINA A Knowledge-Based Information Agent with Social Competence and Human Interaction Capabilities H 2020-645012 D 3 . 3 Advanced version of the vocal analysis techniques in KRISTINA

  2017influential citation
• Transition-Based Dependency Parsing with Stack Long Short-Term Memory

  2015cites this paper