Parsing as Language Modeling

We recast syntactic parsing as a language modeling problem and use recent advances in neural network language modeling to achieve a new state of the art for constituency Penn Treebank parsing — 93.8 F 1 on section 23, us-ing 2-21 as training, 24 as development, plus tri-training. When trees are converted to Stanford dependencies, UAS and LAS are 95.9% and 94.1%.

• Publication year

  2016

• Venue

  Conference on Empirical Methods in Natural Language Processing

• Publication date

  2016-11-01

• Fields of study

  Linguistics, Computer Science

• Identifiers
  DOI 10.18653/v1/D16-1257
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Globally Normalized Transition-Based Neural Networks

  2016influential reference
• Recurrent Neural Network Grammars

  2016influential reference
• Exploring the Limits of Language Modeling

  2016influential reference
• Character-Aware Neural Language Models

  2015influential reference
• Multi-task Sequence to Sequence Learning

  2015influential reference
• An Empirical Exploration of Recurrent Network Architectures

  2015cited by this paper
• Syntactic Parse Fusion

  2015cited by this paper
• Grammar as a Foreign Language

  2014influential reference
• An Empirical Comparison of Parsing Methods for Stanford Dependencies

  2014cited by this paper
• Ambiguity-aware Ensemble Training for Semi-supervised Dependency Parsing

  2014cited by this paper
• Recurrent Neural Network Regularization

  2014influential reference
• Generating Sequences With Recurrent Neural Networks

  2013cited by this paper
• Dropout Improves Recurrent Neural Networks for Handwriting Recognition

  2013cited by this paper
• Fast and Accurate Shift-Reduce Constituent Parsing

  2013cited by this paper
• Training Recurrent Neural Networks

  2013influential reference
• Noise-Contrastive Estimation of Unnormalized Statistical Models, with Applications to Natural Image Statistics

  2012cited by this paper
• On the difficulty of training recurrent neural networks

  2012cited by this paper
• Bayesian Symbol-Refined Tree Substitution Grammars for Syntactic Parsing

  2012cited by this paper
• On the training of recurrent neural networks

  2011cited by this paper
• Self-Training with Products of Latent Variable Grammars

  2010cited by this paper
• Products of Random Latent Variable Grammars

  2010cited by this paper
• Effective Self-Training for Parsing

  2006cited by this paper
• Hierarchical Probabilistic Neural Network Language Model

  2005cited by this paper
• Learning Precise Timing with LSTM Recurrent Networks

  2003cited by this paper
• A Maximum-Entropy-Inspired Parser

  2000cited by this paper
• Long Short-Term Memory

  1997cited by this paper
• Building a Large Annotated Corpus of English: The Penn Treebank

  1993cited by this paper

• Lexicalized Constituency Parsing for Middle Dutch: Low-resource Training and Cross-Domain Generalization

  2026cites this paper
• Active use of latent tree-structured sentence representation in humans and large language models

  2025cites this paper
• A Systematic Study of Compositional Syntactic Transformer Language Models

  2025cites this paper
• If Attention Serves as a Cognitive Model of Human Memory Retrieval, What is the Plausible Memory Representation?

  2025cites this paper
• Structured Tree Alignment for Evaluation of (Speech) Constituency Parsing

  2024cites this paper
• Emergent Word Order Universals from Cognitively-Motivated Language Models

  2024cites this paper
• Hierarchical syntactic structure in human-like language models

  2024cites this paper
• Dependency Transformer Grammars: Integrating Dependency Structures into Transformer Language Models

  2024cites this paper
• Generative Pretrained Structured Transformers: Unsupervised Syntactic Language Models at Scale

  2024cites this paper
• Explicitly Representing Syntax Improves Sentence-to-Layout Prediction of Unexpected Situations

  2024cites this paper
• Pushdown Layers: Encoding Recursive Structure in Transformer Language Models

  2023cites this paper
• Intelligent Assistant Language Understanding On Device

  2023cites this paper
• Context-Free Grammars and Constituency Parsing

  2023cites this paper
• SynJax: Structured Probability Distributions for JAX

  2023cites this paper
• Prompt Design and Answer Processing for Knowledge Base Construction from Pre-trained Language Models

  2022cites this paper
• Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scale

  2022cites this paper
• Aspect-based Sentiment Analysis with Opinion Tree Generation

  2022cites this paper
• Composition, Attention, or Both?

  2022cites this paper
• Exploring Pre-Trained Language Models to Build Knowledge Graph for Metal-Organic Frameworks (MOFs)

  2022cites this paper
• N-ary Constituent Tree Parsing with Recursive Semi-Markov Model

  2021cites this paper
• Linguistic Frameworks Go Toe-to-Toe at Neuro-Symbolic Language Modeling

  2021cites this paper
• A Language Model-based Generative Classifier for Sentence-level Discourse Parsing

  2021cites this paper
• Paradigm Shift in Natural Language Processing

  2021cites this paper
• Neurocomputational Models of Language Processing

  2021cites this paper
• Structural Guidance for Transformer Language Models

  2021cites this paper
• The Limitations of Limited Context for Constituency Parsing

  2021cites this paper
• Structured Prediction as Translation between Augmented Natural Languages

  2021cites this paper
• Improving Disfluency Detection by Self-Training a Self-Attentive Model

  2020cites this paper
• Disentangling Adaptive Gradient Methods from Learning Rates

  2020cites this paper
• Strongly Incremental Constituency Parsing with Graph Neural Networks

  2020cites this paper
• Improving Constituency Parsing with Span Attention

  2020cites this paper
• Structural Supervision Improves Few-Shot Learning and Syntactic Generalization in Neural Language Models

  2020cites this paper
• A Survey of the Usages of Deep Learning for Natural Language Processing

  2020cites this paper
• Improving AMR Parsing with Sequence-to-Sequence Pre-training

  2020cites this paper
• The Influence of Structural Information on Natural Language Processing

  2020cites this paper
• Syntax Role for Neural Semantic Role Labeling

  2020cites this paper
• Memory-bounded Neural Incremental Parsing for Psycholinguistic Prediction

  2020cites this paper
• A survey of syntactic-semantic parsing based on constituent and dependency structures

  2020cites this paper
• Evaluating Approaches to Personalizing Language Models

  2020cites this paper
• Syntactic Structure Distillation Pretraining for Bidirectional Encoders

  2020cites this paper
• Self Attended Stack-Pointer Networks for Learning Long Term Dependencies

  2020cites this paper
• A Span-based Linearization for Constituent Trees

  2020cites this paper
• Improving Molecular Design by Stochastic Iterative Target Augmentation

  2020cites this paper
• Multi-task and multi-view training for end-to-end relation extraction

  2019cites this paper
• Structural Supervision Improves Learning of Non-Local Grammatical Dependencies

  2019influential citation
• Unlexicalized Transition-based Discontinuous Constituency Parsing

  2019cites this paper
• Path-Based Neural Constituency Parsing by Katia Patkin

  2019cites this paper
• Syntax-Infused Variational Autoencoder for Text Generation

  2019cites this paper
• Hierarchical Representation in Neural Language Models: Suppression and Recovery of Expectations

  2019cites this paper
• Scalable Syntax-Aware Language Models Using Knowledge Distillation

  2019cites this paper
• Head-Driven Phrase Structure Grammar Parsing on Penn Treebank

  2019cites this paper
• Cross-Domain Generalization of Neural Constituency Parsers

  2019cites this paper
• Language Modeling with Shared Grammar

  2019cites this paper
• Sentence Encoder Sentence Decoder Tree Encoder Tree Decoder Sentence Vocabulary Tree Vocabulary MLP MLP N μ 2 σ 2

  2019cites this paper
• Modeling Source Syntax and Semantics for Neural AMR Parsing

  2019cites this paper
• Establishing Strong Baselines for the New Decade: Sequence Tagging, Syntactic and Semantic Parsing with BERT

  2019cites this paper
• Concurrent Parsing of Constituency and Dependency

  2019cites this paper
• Incorporating Source Syntax into Transformer-Based Neural Machine Translation

  2019cites this paper
• A Comparative Study on Parsing in Natural Language Processing

  2019cites this paper
• Representation of Constituents in Neural Language Models: Coordination Phrase as a Case Study

  2019cites this paper
• Neural Generative Rhetorical Structure Parsing

  2019cites this paper
• Span-based Hierarchical Semantic Parsing for Task-Oriented Dialog

  2019cites this paper
• Variable beam search for generative neural parsing and its relevance for the analysis of neuro-imaging signal

  2019cites this paper
• Hierarchical Structural Learning for Language Generation

  2019cites this paper
• Moving beyond parallel data for neutral machine translation

  2019cites this paper
• A General-Purpose Algorithm for Constrained Sequential Inference

  2019cites this paper
• Measuring Generalization and Overfitting in Machine Learning

  2019cites this paper
• Understanding EFL Linguistic Models through Relationship between Natural Language Processing and Artificial Intelligence Applications

  2019cites this paper
• Rethinking Self-Attention: Towards Interpretability in Neural Parsing

  2019cites this paper
• Grammaticality in neural natural language generation

  2019cites this paper
• Syntax-Aware Language Modeling with Recurrent Neural Networks

  2018cites this paper
• Improving Sequence-to-Sequence Constituency Parsing

  2018cites this paper
• Semi-Supervised Sequence Modeling with Cross-View Training

  2018cites this paper
• Annotating and Modeling Shallow Semantics Directly from Text

  2018influential citation
• 2 Previous Work 2 . 1 Neural Language Modeling with LSTMs

  2018cites this paper
• Sequential Parsing with In-order Tree Traversals

  2018influential citation
• Constituency Parsing with a Self-Attentive Encoder

  2018cites this paper
• Top-down Tree Structured Decoding with Syntactic Connections for Neural Machine Translation and Parsing

  2018cites this paper
• What’s Going On in Neural Constituency Parsers? An Analysis

  2018cites this paper
• Direct Output Connection for a High-Rank Language Model

  2018cites this paper
• Multi-Source Syntactic Neural Machine Translation

  2018cites this paper
• Two Local Models for Neural Constituent Parsing

  2018influential citation
• An Empirical Study of Building a Strong Baseline for Constituency Parsing

  2018cites this paper
• LSTMs Can Learn Syntax-Sensitive Dependencies Well, But Modeling Structure Makes Them Better

  2018influential citation
• A Survey of the Usages of Deep Learning in Natural Language Processing

  2018cites this paper
• An Improved Neural Network Model for Joint POS Tagging and Dependency Parsing

  2018cites this paper
• Finding syntax in human encephalography with beam search

  2018cites this paper
• Policy Gradient as a Proxy for Dynamic Oracles in Constituency Parsing

  2018cites this paper
• On the Practical Computational Power of Finite Precision RNNs for Language Recognition

  2018cites this paper
• Multilingual Constituency Parsing with Self-Attention and Pre-Training

  2018cites this paper
• Improved Dependency Parsing using Implicit Word Connections Learned from Unlabeled Data

  2018cites this paper
• Cross-lingual Decompositional Semantic Parsing

  2018cites this paper
• Constituent Parsing as Sequence Labeling

  2018cites this paper
• Linear-time Constituency Parsing with RNNs and Dynamic Programming

  2018cites this paper
• MT/IE: Cross-lingual Open Information Extraction with Neural Sequence-to-Sequence Models

  2017cites this paper
• Improving Feature-Rich Transition-Based Constituent Parsing Using Recurrent Neural Networks

  2017cites this paper
• Effective Inference for Generative Neural Parsing

  2017influential citation
• Extraction with Neural Sequence-to-Sequence Models

  2017cites this paper
• Deep Semantic Role Labeling: What Works and What’s Next

  2017cites this paper
• In-Order Transition-based Constituent Parsing

  2017influential citation