Analyzing Optimization for Statistical Machine Translation: MERT Learns Verbosity, PRO Learns Length

We study the impact of source length and verbosity of the tuning dataset on the performance of parameter optimizers such as MERT and PRO for statistical machine translation. In particular, we test whether the verbosity of the resulting translations can be modified by varying the length or the verbosity of the tuning sentences. We find that MERT learns the tuning set verbosity very well, while PRO is sensitive to both the verbosity and the length of the source sentences in the tuning set; yet, overall PRO learns best from highverbosity tuning datasets. Given these dependencies, and potentially some other such as amount of reordering, number of unknown words, syntactic complexity, and evaluation measure, to mention just a few, we argue for the need of controlled evaluation scenarios, so that the selection of tuning set and optimization strategy does not overshadow scientific advances in modeling or decoding. In the mean time, until we develop such controlled scenarios, we recommend using PRO with a large verbosity tuning set, which, in our experiments, yields highest BLEU across datasets and language pairs.

• Publication year

  2015

• Venue

  Conference on Computational Natural Language Learning

• Publication date

  2015-07-01

• Fields of study

  Linguistics, Computer Science

• Identifiers
  DOI 10.18653/v1/K15-1007
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• APRO: All-Pairs Ranking Optimization for MT Tuning

  2015cited by this paper
• A Tale about PRO and Monsters

  2013cited by this paper
• Structured Ramp Loss Minimization for Machine Translation

  2012influential reference
• Optimizing for Sentence-Level BLEU+1 Yields Short Translations

  2012influential reference
• Locally Training the Log-Linear Model for SMT

  2012cited by this paper
• Tuning as Linear Regression

  2012cited by this paper
• Batch Tuning Strategies for Statistical Machine Translation

  2012influential reference
• Joint Feature Selection in Distributed Stochastic Learning for Large-Scale Discriminative Training in SMT

  2012influential reference
• Generalization bounds and consistency for latent-structural probit and ramp loss

  2011cited by this paper
• Better Hypothesis Testing for Statistical Machine Translation: Controlling for Optimizer Instability

  2011cited by this paper
• Methods for Smoothing the Optimizer Instability in SMT

  2011cited by this paper
• Tuning as Ranking

  2011cited by this paper
• KenLM: Faster and Smaller Language Model Queries

  2011cited by this paper
• Adaptive Development Data Selection for Log-linear Model in Statistical Machine Translation

  2010cited by this paper
• cdec: A Decoder, Alignment, and Learning Framework for Finite- State and Context-Free Translation Models

  2010cited by this paper
• Domain adaptation for statistical machine translation in development corpus selection

  2010cited by this paper
• Jane: Open Source Hierarchical Translation, Extended with Reordering and Lexicon Models

  2010cited by this paper
• Joshua: An Open Source Toolkit for Parsing-Based Machine Translation

  2009cited by this paper
• 11,001 New Features for Statistical Machine Translation

  2009cited by this paper
• The Meteor metric for automatic evaluation of machine translation

  2009cited by this paper
• Online Large-Margin Training of Syntactic and Structural Translation Features

  2008cited by this paper
• Regularization and Search for Minimum Error Rate Training

  2008cited by this paper
• Random Restarts in Minimum Error Rate Training for Statistical Machine Translation

  2008cited by this paper
• Arabic Morphological Tagging, Diacritization, and Lemmatization Using Lexeme Models and Feature Ranking

  2008cited by this paper
• Moses: Open Source Toolkit for Statistical Machine Translation

  2007influential reference
• Online Large-Margin Training for Statistical Machine Translation

  2007influential reference
• A Study of Translation Edit Rate with Targeted Human Annotation

  2006cited by this paper
• Clause Restructuring for Statistical Machine Translation

  2005cited by this paper
• Dependency Treelet Translation: Syntactically Informed Phrasal SMT

  2005cited by this paper
• A Hierarchical Phrase-Based Model for Statistical Machine Translation

  2005cited by this paper
• Edinburgh System Description for the 2005 IWSLT Speech Translation Evaluation

  2005cited by this paper
• A STUDY OF TRANSLATION ERROR RATE WITH TARGETED HUMAN ANNOTATION

  2005cited by this paper
• What’s in a translation rule?

  2004cited by this paper
• Statistical Phrase-Based Translation

  2003cited by this paper
• Minimum Error Rate Training in Statistical Machine Translation

  2003influential reference
• Bleu: a Method for Automatic Evaluation of Machine Translation

  2002cited by this paper

• Automatic Document Sketching: Generating Drafts from Analogous Texts

  2021cites this paper
• Top-Rank Enhanced Listwise Optimization for Statistical Machine Translation

  2017cites this paper
• Robust Tuning Datasets for Statistical Machine Translation

  2017cites this paper
• The MSR-NLP System at Dialog System Technology Challenges 6

  2017cites this paper
• Bi-text Alignment of Movie Subtitles for Spoken English-Arabic Statistical Machine Translation

  2016cites this paper
• Research on Intelligent Automatic Translation System in Chinese and English Based on Integration Technology

  2016cites this paper