Labelled Dependencies in Machine Translation Evaluation

We present a method for evaluating the quality of Machine Translation (MT) output, using labelled dependencies produced by a Lexical-Functional Grammar (LFG) parser. Our dependency-based method, in contrast to most popular string-based evaluation metrics, does not unfairly penalize perfectly valid syntactic variations in the translation, and the addition of WordNet provides a way to accommodate lexical variation. In comparison with other metrics on 16,800 sentences of Chinese-English newswire text, our method reaches high correlation with human scores.

• Publication year

  2007

• Venue

  WMT@ACL

• Publication date

  2007-06-23

• Fields of study

  Linguistics, Computer Science

• Identifiers
  DOI 10.3115/1626355.1626369
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Paraphrasing for Automatic Evaluation

  2006influential reference
• CDER: Efficient MT Evaluation Using Block Movements

  2006cited by this paper
• Contextual Bitext-Derived Paraphrases in Automatic MT Evaluation

  2006influential reference
• Re-evaluating the Role of Bleu in Machine Translation Research

  2006cited by this paper
• A Paraphrase-Based Approach to Machine Translation Evaluation

  2005cited by this paper
• Syntactic Features for Evaluation of Machine Translation

  2005influential reference
• METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments

  2005influential reference
• A STUDY OF TRANSLATION ERROR RATE WITH TARGETED HUMAN ANNOTATION

  2005influential reference
• Europarl: A Parallel Corpus for Statistical Machine Translation

  2005cited by this paper
• Measuring confidence intervals for the machine translation evaluation metrics

  2004cited by this paper
• A learning approach to improving sentence-level MT evaluation

  2004cited by this paper
• Lexical Functional Grammar A Formal System for Grammatical Representation

  2004cited by this paper
• Pharaoh: A Beam Search Decoder for Phrase-Based Statistical Machine Translation Models

  2004cited by this paper
• Long-Distance Dependency Resolution in Automatically Acquired Wide-Coverage PCFG-Based LFG Approximations

  2004influential reference
• Evaluation of machine translation and its evaluation

  2003influential reference
• A Systematic Comparison of Various Statistical Alignment Models

  2003cited by this paper
• Head-Driven Statistical Models for Natural Language Parsing

  2003cited by this paper
• Automatic evaluation of machine translation quality using n-gram co-occurrence statistics

  2002influential reference
• Bleu: a Method for Automatic Evaluation of Machine Translation

  2002influential reference
• Lexical-Functional Syntax

  2000cited by this paper
• The Mental Representation of Grammatical Relations

  1986cited by this paper
• Conference

  1969cited by this paper
• REPRESENTATION

  1919cited by this paper

• Constructing the Evaluation Index System of Chinese-Portuguese Machine Translation using the Delphi and Analytic Hierarchy Process Methods

  2023cites this paper
• Machine Translation Evaluation Metric Based on Dependency Parsing Model

  2019cites this paper
• Estimating word-level quality of statistical machine translation output using monolingual information alone

  2019cites this paper
• Research on Machine Translation Automatic Evaluation Based on Extended Reference

  2019cites this paper
• Quality expectations of machine translation

  2018cites this paper
• VERTa: a linguistic approach to automatic machine translation evaluation

  2018cites this paper
• LEPOR: An Augmented Machine Translation Evaluation Metric

  2017cites this paper
• The mat sat on the cat: investigating structure in the evaluation of order in machine translation

  2017cites this paper
• ENTF: An Entropy-Based MT Evaluation Metric

  2017cites this paper
• Medical Data Machine Translation Evaluation Based on Dependency N-Grams

  2016cites this paper
• Through the Eyes of VERTa

  2016cites this paper
• DTED: Evaluation of Machine Translation Structure Using Dependency Parsing and Tree Edit Distance

  2016cites this paper
• Guiding automatic MT evaluation by means of linguistic features

  2016cites this paper
• Book Review: Syntax-Based Statistical Machine Translation by Philip Williams, Rico Sennrich, Matt Post and Philipp Koehn

  2016cites this paper
• Automatic Machine Translation Evaluation: A Qualitative Approach

  2015cites this paper
• VERTa: a Linguistically-motivated Metric at the WMT15 Metrics Task

  2015cites this paper
• Improve the Evaluation of Fluency Using Entropy for Machine Translation Evaluation Metrics

  2015cites this paper
• Targeted Paraphrasing on Deep Syntactic Layer for MT Evaluation

  2015cites this paper
• Improve the Evaluation of Translation Fluency by Using Entropy of Matched Sub-segments

  2015cites this paper
• Targeted Paraphrasing of Czech Sentences for Machine Translation Evaluation

  2015cites this paper
• RED, The DCU-CASICT Submission of Metrics Tasks

  2014cites this paper
• BiMEANT: Integrating Cross-Lingual and Monolingual Semantic Frame Similarities in the MEANT Semantic MT Evaluation Metric

  2014cites this paper
• VERTa participation in the WMT14 Metrics Task

  2014cites this paper
• DCU Participation in WMT2013 Metrics Task

  2013cites this paper
• CCG-augmented hierarchical phrase-based statistical machinetranslation

  2013cites this paper
• Quality Estimation for Machine Translation Using the Joint Method of Evaluation Criteria and Statistical Modeling

  2013cites this paper
• Learning to Translate with Multiple Objectives

  2012cites this paper
• DCU-Symantec Submission for the WMT 2012 Quality Estimation Task

  2012cites this paper
• Use and Evaluation of Controlled Languages in Industrial Environments and Feasibility Study for the Implementation of Machine Translation

  2012cites this paper
• Improving Recombination in a Linear EBMT System by Use of Constraints

  2012cites this paper
• VERTa: Linguistic features in MT evaluation

  2012cites this paper
• Lost and Found in Translation: Cross-Lingual Question Answering with Result Translation

  2012cites this paper
• Corroborating Text Evaluation Results with Heterogeneous Measures

  2011cites this paper
• Using metrics from complex networks to evaluate machine translation

  2011cites this paper
• Consistent Translation using Discriminative Learning - A Translation Memory-inspired Approach

  2011cites this paper
• The Integration of Machine Translation and Translation Memory

  2011influential citation
• Chapter 5: Machine Translation Evaluation and Optimization

  2011cites this paper
• VERTa : Exploring a Multidimensional Linguistically-Motivated Metric

  2011cites this paper
• Machine Translation Evaluation and Optimization

  2011cites this paper
• Automatic Evaluation in Machine Translation Towards Similarity Measures Based on Multiple Linguistic Layers

  2010cites this paper
• The DCU Dependency-Based Metric in WMT-MetricsMATR 2010

  2010influential citation
• All in Strings: a Powerful String-based Automatic MT Evaluation Metric with Multiple Granularities

  2010cites this paper
• An algorithm for cross-lingual sense-clustering tested in a MT evaluation setting

  2010cites this paper
• Adapting Stochastic Output for Rule-Based Semantics Wissenschaftliche

  2010cites this paper
• Metric and reference factors in minimum error rate training

  2010cites this paper
• Data-Driven Semantic Analysis for Multilingual WSD and Lexical Selection in Translation

  2009cites this paper
• Learning Labelled Dependencies in Machine Translation Evaluation

  2009influential citation
• The Contribution of Linguistic Features to Automatic Machine Translation Evaluation

  2009cites this paper
• Improving the Objective Function in Minimum Error Rate Training

  2009cites this paper
• u´õstico en Mde Evaluacion Automde Traducciones Linguistic Processing in Automatic Translation Evaluation Metrics

  2009cites this paper
• Expected dependency pair match: predicting translation quality with expected syntactic structure

  2009cites this paper
• Universidad Politécnica de Madrid

  2009cites this paper
• Capturing Lexical Variation in MT Evaluation Using Automatically Built Sense-Cluster Inventories

  2009cites this paper
• Symbolic-to-statistical hybridization: extending generation-heavy machine translation

  2009cites this paper
• Integrating Linguistic Information in Machine Translation Evaluation

  2008cites this paper
• A novel dependency-based evaluation metric for machine translation

  2008cites this paper
• Dependency Structure Trees in Syntax Based Machine Translation

  2008cites this paper
• Automatic Syntactic MT Evaluation with Expected Dependency Pair Match

  2008cites this paper
• S EPIA : Surface Span Extension to Syntactic Dependency Precision-based MT Evaluation

  2008cites this paper
• Empirical machine translation and its evaluation

  2008cites this paper
• Data-Driven Machine Translation for Sign Languages

  2008cites this paper
• Evaluating machine translation with LFG dependencies

  2007cites this paper
• Using F-structures in machine translation evaluation

  2007cites this paper
• Automatic Acquisition of Lexical-Functional Grammar Resources from a Japanese Dependency Corpus

  2007cites this paper
• Seventh Framework Programme Theme 3 Information and Communication Technologies Panacea Project Platform for Automatic, Normalized Annotation and Cost-effective Acquisition of Language Resources for Human Language Technologies Criteria for Evaluation of Resources, Technology and Integration

  year unknowncites this paper