Overview of the SPMRL 2013 Shared Task: A Cross-Framework Evaluation of Parsing Morphologically Rich Languages

This paper reports on the first shared task on statistical parsing of morphologically rich languages (MRLs). The task features data sets from nine languages, each available both in constituency and dependency annotation. We report on the preparation of the data sets, on the proposed parsing scenarios, and on the evaluation metrics for parsing MRLs given different representation types. We present and analyze parsing results obtained by the task participants, and then provide an analysis and comparison of the parsers across languages and frameworks, reported for gold input as well as more realistic parsing scenarios.

• Publication year

  2013

• Venue

  SPMRL@EMNLP

• Publication date

  2013-10-18

• Fields of study

  Linguistics, Computer Science

• Identifiers
  DOI 10.18653/v1/w13-4917
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• LDC Arabic Treebanks and Associated Corpora: Data Divisions Manual

  2013cited by this paper
• Exploring beam-based shift-reduce dependency parsing with DyALog: Results from the SPMRL 2013 shared task

  2013cited by this paper
• Parsing Croatian and Serbian by Using Croatian Dependency Treebanks

  2013influential reference
• SPMRL‘13 Shared Task System: The CADIM Arabic Dependency Parser

  2013influential reference
• magyarlanc: A Tool for Morphological and Dependency Parsing of Hungarian

  2013cited by this paper
• Universal Dependency Annotation for Multilingual Parsing

  2013cited by this paper
• A Unified Morpho-Syntactic Scheme of Stanford Dependencies

  2013cited by this paper
• Parsing Morphologically Rich Languages: Introduction to the Special Issue

  2013influential reference
• Dependency Parsing of Modern Standard Arabic with Lexical and Inflectional Features

  2013influential reference
• The LIGM-Alpage architecture for the SPMRL 2013 Shared Task: Multiword Expression Analysis and Dependency Parsing

  2013influential reference
• Effective Morphological Feature Selection with MaltOptimizer at the SPMRL 2013 Shared Task

  2013influential reference
• Knowledge Sources for Constituent Parsing of German, a Morphologically Rich and Less-Configurational Language

  2013cited by this paper
• Preparing Korean Data for the Shared Task on Parsing Morphologically Rich Languages

  2013cited by this paper
• The AI-KU System at the SPMRL 2013 Shared Task : Unsupervised Features for Dependency Parsing

  2013influential reference
• Parsing Models for Identifying Multiword Expressions

  2013cited by this paper
• Exploiting the Contribution of Morphological Information to Parsing: the BASQUE TEAM system in the SPRML‘2013 Shared Task

  2013influential reference
• (Re)ranking Meets Morphosyntax: State-of-the-art Results from the SPMRL 2013 Shared Task

  2013influential reference
• Statistical Parsing of Spanish and Data Driven Lemmatization

  2012cited by this paper
• Bayesian Symbol-Refined Tree Substitution Grammars for Syntactic Parsing

  2012cited by this paper
• Polish Dependency Bank

  2012cited by this paper
• Using an SVM Ensemble System for Improved Tamil Dependency Parsing

  2012cited by this paper
• Cross-Framework Evaluation for Statistical Parsing

  2012cited by this paper
• Making Ellipses Explicit in Dependency Conversion for a German Treebank

  2012cited by this paper
• Detecting grammatical errors with treebank-induced, probabilistic parsers

  2012cited by this paper
• PLCFRS Parsing Revisited: Restricting the Fan-Out to Two

  2012cited by this paper
• A Transition-Based System for Joint Part-of-Speech Tagging and Labeled Non-Projective Dependency Parsing

  2012cited by this paper
• Overview of the 2012 Shared Task on Parsing the Web

  2012cited by this paper
• MaltOptimizer: An Optimization Tool for MaltParser

  2012cited by this paper
• Transition-based Dependency Parsing with Rich Non-local Features

  2011cited by this paper
• A Preliminary Version of Składnica — a Treebank of Polish

  2011cited by this paper
• Coarse-to-Fine Natural Language Processing

  2011cited by this paper
• Statistical Dependency Parsing in Korean: From Corpus Generation To Automatic Parsing

  2011cited by this paper
• Data point selection for self-training

  2011cited by this paper
• French parsing enhanced with a word clustering method based on a syntactic lexicon

  2011cited by this paper
• Data-driven Parsing using PLCFRS Data-driven Parsing using Probabilistic Linear Context-Free Rewriting Systems

  2011cited by this paper
• Factors Affecting the Accuracy of Korean Parsing

  2010influential reference
• Application of Different Techniques to Dependency Parsing of Basque

  2010cited by this paper
• An Efficient Algorithm for Easy-First Non-Directional Dependency Parsing

  2010cited by this paper
• Turbo Parsers: Dependency Parsing by Approximate Variational Inference

  2010cited by this paper
• Hungarian Dependency Treebank

  2010cited by this paper
• Better Arabic Parsing: Baselines, Evaluations, and Analysis

  2010influential reference
• A Morphosyntactic Brill Tagger for Inflectional Languages

  2010cited by this paper
• Statistical French Dependency Parsing: Treebank Conversion and First Results

  2010cited by this paper
• Self-Training with Products of Latent Variable Grammars

  2010cited by this paper
• Top Accuracy and Fast Dependency Parsing is not a Contradiction

  2010influential reference
• Easy-First Dependency Parsing of Modern Hebrew

  2010cited by this paper
• Towards a Bank of Constituent Parse Trees for Polish

  2010cited by this paper
• On the Role of Morphosyntactic Features in Hindi Dependency Parsing

  2010cited by this paper
• Handling Unknown Words in Statistical Latent-Variable Parsing Models for Arabic, English and French

  2010cited by this paper
• Modeling Morphosyntactic Agreement in Constituency-Based Parsing of Modern Hebrew

  2010cited by this paper
• Statistical Parsing of Morphologically Rich Languages (SPMRL) What, How and Whither

  2010influential reference
• Parsing Word Clusters

  2010cited by this paper
• Products of Random Latent Variable Grammars

  2010cited by this paper
• Lemmatization and Statistical Lexicalized Parsing of Morphologically-Rich Languages

  2010cited by this paper
• Enhancing Unlexicalized Parsing Performance Using a Wide Coverage Lexicon, Fuzzy Tag-Set Mapping, and EM-HMM-Based Lexical Probabilities

  2009cited by this paper
• CATiB: The Columbia Arabic Treebank

  2009influential reference
• MADA + TOKAN : A Toolkit for Arabic Tokenization , Diacritization , Morphological Disambiguation , POS Tagging , Stemming and Lemmatization

  2009influential reference
• Syntactic Annotation in the Columbia Arabic Treebank

  2009influential reference
• Efficient, Feature-based, Conditional Random Field Parsing

  2008cited by this paper
• Forest Reranking: Discriminative Parsing with Non-Local Features

  2008cited by this paper
• From Dependencies to Constituents in the Reference Corpus for the Processing of Basque (EPEC)

  2008cited by this paper
• TAG, Dynamic Programming, and the Perceptron for Efficient, Feature-Rich Parsing

  2008cited by this paper
• Unsupervised Lexicon-Based Resolution of Unknown Words for Full Morphological Analysis

  2008cited by this paper
• The PaGe 2008 Shared Task on Parsing German

  2008cited by this paper
• The Stanford Typed Dependencies Representation

  2008cited by this paper
• Expériences d’analyse syntaxique statistique du français

  2008cited by this paper
• Language resources for Hebrew

  2008cited by this paper
• How to Compare Treebanks

  2008cited by this paper
• Automatic Annotation of Morpho-Syntactic Dependencies in a Modern Hebrew Treebank

  2008cited by this paper
• Learning Morphology with Morfette

  2008influential reference
• From Dependencies to Constituents in the Reference Corpus for the Processing of Basque (EPEC)

  2008cited by this paper
• EM Can Find Pretty Good HMM POS-Taggers (When Given a Good Start)

  2008cited by this paper
• MaltParser: A Language-Independent System for Data-Driven Dependency Parsing

  2007influential reference
• Single Malt or Blended? A Study in Multilingual Parser Optimization

  2007cited by this paper
• The CoNLL 2007 Shared Task on Dependency Parsing

  2007cited by this paper
• Treebank Annotation Schemes and Parser Evaluation for German

  2007cited by this paper
• Determining Case in Arabic: Learning Complex Linguistic Behavior Requires Complex Linguistic Features

  2007cited by this paper
• MaltParser: A language-independent system for data-driven dependency parsing

  2007cited by this paper
• Evaluating Evaluation Measures

  2007cited by this paper
• Discontinuity Revisited: An Improved Conversion to Context-Free Representations

  2007influential reference
• Preparing, restructuring, and augmenting a French treebank:lexicalised parsers or coherent treebanks?

  2007cited by this paper
• Bootstrapping a Swedish Treebank Using Cross-Corpus Harmonization and Annotation Projection

  2007cited by this paper
• HunPos: an open source trigram tagger

  2007cited by this paper
• Parser Combination by Reparsing

  2006cited by this paper
• Learning Accurate, Compact, and Interpretable Tree Annotation

  2006influential reference
• Effective Self-Training for Parsing

  2006cited by this paper
• CoNLL-X Shared Task on Multilingual Dependency Parsing

  2006influential reference
• Is it Really that Difficult to Parse German?

  2006cited by this paper
• Parsing the Arabic Treebank: Analysis and Improvements

  2006influential reference
• An Unsupervised Morpheme-Based HMM for Hebrew Morphological Disambiguation

  2006cited by this paper
• Talbanken05: A Swedish Treebank with Phrase Structure and Dependency Annotation

  2006cited by this paper
• Coarse-to-Fine n-Best Parsing and MaxEnt Discriminative Reranking

  2005cited by this paper
• A survey on tree edit distance and related problems

  2005influential reference
• Online Large-Margin Training of Dependency Parsers

  2005cited by this paper
• The Szeged Treebank

  2005cited by this paper
• Analyzing an Italian Treebank with State-of-the-Art Statistical Parsers

  2004cited by this paper
• The Penn Arabic Treebank: Building a Large-Scale Annotated Arabic Corpus

  2004influential reference
• SMOR: A German Computational Morphology Covering Derivation, Composition and Inflection

  2004cited by this paper
• Head-Driven Statistical Models for Natural Language Parsing

  2003cited by this paper
• Construction of a Basque Dependency Treebank

  2003cited by this paper

• Self-Correction Makes LLMs Better Parsers

  2025cites this paper
• Construction and educational application of a linguistically grounded dependency treebank for Uyghur

  2025cites this paper
• SELEXINI - a large and diverse automatically parsed corpus of French

  2025influential citation
• Constituency Structure over Eojeol in Korean Treebanks

  2025cites this paper
• Parallel Trees: a novel resource with aligned dependency and constituency syntactic representations

  2025cites this paper
• NLPre: A Revised Approach towards Language-centric Benchmarking of Natural Language Preprocessing Systems

  2024cites this paper
• Structured Tree Alignment for Evaluation of (Speech) Constituency Parsing

  2024cites this paper
• K-UD: Revising Korean Universal Dependencies Guidelines

  2024cites this paper
• Automating Easy Read Text Segmentation

  2024cites this paper
• From Partial to Strictly Incremental Constituent Parsing

  2024cites this paper
• jp-evalb: Robust Alignment-based PARSEVAL Measures

  2024cites this paper
• Parsing Headed Constituencies

  2024cites this paper
• Unsupervised Parsing by Searching for Frequent Word Sequences among Sentences with Equivalent Predicate-Argument Structures

  2024influential citation
• CroissantLLM: A Truly Bilingual French-English Language Model

  2024cites this paper
• Improving Unsupervised Constituency Parsing via Maximizing Semantic Information

  2024influential citation
• Parsing Early New High German: Benefits and limitations of cross-dialectal training

  2023cites this paper
• Unsupervised and Few-Shot Parsing from Pretrained Language Models (Extended Abstract)

  2023cites this paper
• A Novel Alignment-based Approach for PARSEVAL Measuress

  2023cites this paper
• LLM-enhanced Self-training for Cross-domain Constituency Parsing

  2023cites this paper
• Attention sur les spans pour l’analyse syntaxique en constituants

  2023influential citation
• An Exploration of Left-Corner Transformations

  2023cites this paper
• ICON: Building a Large-Scale Benchmark Constituency Treebank for the Indonesian Language

  2023cites this paper
• Constituency Parsing with Spines and Attachments

  2023cites this paper
• Don’t Parse, Choose Spans! Continuous and Discontinuous Constituency Parsing via Autoregressive Span Selection

  2023cites this paper
• Contextual Distortion Reveals Constituency: Masked Language Models are Implicit Parsers

  2023cites this paper
• Approximating CKY with Transformers

  2023cites this paper
• ICON: A Linguistically-Motivated Large-Scale Benchmark Indonesian Constituency Treebank

  2023cites this paper
• Improving the Extraction of Supertags for Constituency Parsing with Linear Context-Free Rewriting Systems

  2022cites this paper
• Reorder and then Parse, Fast and Accurate Discontinuous Constituency Parsing

  2022cites this paper
• Large Pre-Trained Models with Extra-Large Vocabularies: A Contrastive Analysis of Hebrew BERT Models and a New One to Outperform Them All

  2022cites this paper
• Unsupervised and few-shot parsing from pretrained language models

  2022influential citation
• A survey on syntactic processing techniques

  2022cites this paper
• AlephBERT: Language Model Pre-training and Evaluation from Sub-Word to Sentence Level

  2022cites this paper
• Breaking Character: Are Subwords Good Enough for MRLs After All?

  2022cites this paper
• Analysing Syntactic and Semantic Features in Pre-trained Language Models in a Fully Unsupervised Setting

  2022cites this paper
• On Parsing as Tagging

  2022cites this paper
• Machine Learning and Hebrew NLP for Automated Assessment of Open-Ended Questions in Biology

  2022cites this paper
• Revisiting the Practical Effectiveness of Constituency Parse Extraction from Pre-trained Language Models

  2022cites this paper
• PAGnol: An Extra-Large French Generative Model

  2021cites this paper
• ParaShoot: A Hebrew Question Answering Dataset

  2021cites this paper
• Investigating Non-local Features for Neural Constituency Parsing

  2021cites this paper
• Annotations Matter: Leveraging Multi-task Learning to Parse UD and SUD

  2021cites this paper
• ELECTRA for Neural Coreference Resolution in Italian

  2021cites this paper
• BERT-Proof Syntactic Structures: Investigating Errors in Discontinuous Constituency Parsing

  2021cites this paper
• COMBO: A New Module for EUD Parsing

  2021cites this paper
• Discontinuous Grammar as a Foreign Language

  2021cites this paper
• A Conditional Splitting Framework for Efficient Constituency Parsing

  2021cites this paper
• Składnica: a constituency treebank of Polish harmonised with the Walenty valency dictionary

  2021cites this paper
• AlephBERT: A Hebrew Large Pre-Trained Language Model to Start-off your Hebrew NLP Application With

  2021cites this paper
• Reducing Discontinuous to Continuous Parsing with Pointer Network Reordering

  2021cites this paper
• Re-thinking Supertags in Linear Context-free Rewriting Systems for Constituency Parsing

  2021cites this paper
• A French corpus annotated for multiword expressions and named entities

  2021cites this paper
• What’s in a Span? Evaluating the Creativity of a Span-Based Neural Constituency Parser

  2021cites this paper
• Thematic Fit Bits: Annotation Quality and Quantity Interplay for Event Participant Representation

  2021influential citation
• Proceedings of the Society for Computation in Linguistics Proceedings of the Society for Computation in Linguistics

  2021cites this paper
• Analyse en dépendances du français avec des plongements contextualisés (French dependency parsing with contextualized embeddings)

  2021cites this paper
• Thematic fit bits: Annotation quality and quantity for event participant representation

  2021cites this paper
• Unsupervised Neural Machine Translation with Universal Grammar

  2021cites this paper
• Efficient Constituency Parsing by Pointing

  2020cites this paper
• FlauBERT : des modèles de langue contextualisés pré-entraînés pour le français (FlauBERT : Unsupervised Language Model Pre-training for French)

  2020cites this paper
• Which Dependency Parser to Use for Distributional Semantics in a Specialized Domain?

  2020cites this paper
• Multilingual Zero-shot Constituency Parsing

  2020cites this paper
• From SPMRL to NMRL: What Did We Learn (and Unlearn) in a Decade of Parsing Morphologically-Rich Languages (MRLs)?

  2020cites this paper
• Extrinsic Evaluation of French Dependency Parsers on a Specialized Corpus: Comparison of Distributional Thesauri

  2020cites this paper
• Neural Modeling for Named Entities and Morphology (NEMO2)

  2020cites this paper
• Multilingual Chart-based Constituency Parse Extraction from Pre-trained Language Models

  2020cites this paper
• A note on constituent parsing for Korean

  2020influential citation
• On understanding character-level models for representing morphology

  2020cites this paper
• Discontinuous Constituent Parsing as Sequence Labeling

  2020cites this paper
• Heads-up! Unsupervised Constituency Parsing via Self-Attention Heads

  2020cites this paper
• Multitask Pointer Network for Multi-Representational Parsing

  2020cites this paper
• A Thousand Words are Worth More Than One Recording: NLP Based Speaker Change Point Detection

  2020cites this paper
• It May Be in the Structure, Not the Combinations: Graph Metrics as an Alternative to Statistical Measures in Corpus-Linguistic Research 245

  2020cites this paper
• Supertagging-based Parsing with Linear Context-free Rewriting Systems

  2020cites this paper
• Analyse automatique par transitions pour l'identification des expressions polylexicales. (Automatic transition-based analysis for multiword expression identification)

  2019influential citation
• Un corpus arboré pour le français : le French Treebank [A parsed corpus for French: the French treebank]

  2019cites this paper
• From partial neural graph-based LTAG parsing towards full parsing

  2019influential citation
• FlauBERT: Unsupervised Language Model Pre-training for French

  2019cites this paper
• Discontinuous Constituency Parsing with a Stack-Free Transition System and a Dynamic Oracle

  2019cites this paper
• Comparing linear and neural models for competitive MWE identification

  2019cites this paper
• A New Annotation Scheme for the Sejong Part-of-speech Tagged Corpus

  2019cites this paper
• Neural Lemmatization of Multiword Expressions

  2019cites this paper
• The Impact of Word Representations on Sequential Neural MWE Identification

  2019cites this paper
• Dependency parsing of Polish

  2019cites this paper
• Unlexicalized Transition-based Discontinuous Constituency Parsing

  2019cites this paper
• Statistical MWE-aware parsing

  2019influential citation
• Comparaison qualitative et extrinsèque d’analyseurs syntaxiques du français : confrontation de modèles distributionnels sur un corpus spécialisé (Extrinsic evaluation of French dependency parsers on a specialised corpus : comparison of distributional thesauri )

  2019cites this paper
• Une note sur l’analyse du constituant pour le français (A Note on constituent parsing for French)

  2018cites this paper
• De la constitution d’un corpus arboré à l’analyse syntaxique du serbe [From the constitution of a treebank to the syntactic analysis of the Serbian language]

  2018cites this paper
• Multilingual Constituency Parsing with Self-Attention and Pre-Training

  2018cites this paper
• The Hebrew Universal Dependency Treebank: Past Present and Future

  2018cites this paper
• Semi-Supervised Methods for Out-of-Domain Dependency Parsing

  2018cites this paper
• Training parsers for low-resourced languages : improving cross-lingual transfer with monolingual knowledge. (Apprentissage d'analyseurs syntaxiques pour les langues peu dotées : amélioration du transfert cross-lingue grâce à des connaissances monolingues)

  2018cites this paper
• Differentiable Perturb-and-Parse: Semi-Supervised Parsing with a Structured Variational Autoencoder

  2018cites this paper
• Using Semantics for Granularities of Tokenization

  2018cites this paper
• Preprocessing Does Matter: Parsing Non-Segmented Arabic

  2018influential citation
• Actes de la conférence Traitement Automatique de la Langue Naturelle, TALN 2018

  2018cites this paper
• CoNLL-UL: Universal Morphological Lattices for Universal Dependency Parsing

  2018cites this paper
• Cheating a Parser to Death: Data-driven Cross-Treebank Annotation Transfer

  2018influential citation
• A New Version of the Składnica Treebank of Polish Harmonised with the Walenty Valency Dictionary

  2018cites this paper