Predicting Success in Machine Translation

The performance of machine translation systems varies greatly depending on the source and target languages involved. Determining the contribution of different characteristics of language pairs on system performance is key to knowing what aspects of machine translation to improve and which are irrelevant. This paper investigates the effect of different explanatory variables on the performance of a phrase-based system for 110 European language pairs. We show that three factors are strong predictors of performance in isolation: the amount of reordering, the morphological complexity of the target language and the historical relatedness of the two languages. Together, these factors contribute 75% to the variability of the performance of the system.

• Publication year

  2008

• Venue

  Conference on Empirical Methods in Natural Language Processing

• Publication date

  2008-10-25

• Fields of study

  Linguistics, Computer Science

• Identifiers
  DOI 10.3115/1613715.1613809
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Further Meta-Evaluation of Machine Translation

  2008cited by this paper
• The “Noisier Channel”: Translation from Morphologically Complex Languages

  2007cited by this paper
• Moses: Open Source Toolkit for Statistical Machine Translation

  2007cited by this paper
• Modelling Lexical Redundancy for Machine Translation

  2006cited by this paper
• A Study of Translation Edit Rate with Targeted Human Annotation

  2006cited by this paper
• A STUDY OF TRANSLATION ERROR RATE WITH TARGETED HUMAN ANNOTATION

  2005cited by this paper
• Improving Statistical MT through Morphological Analysis

  2005cited by this paper
• Europarl: A Parallel Corpus for Statistical Machine Translation

  2005influential reference
• The World Atlas of Language Structures

  2005cited by this paper
• The Mathematics of Machine Translation : Parameter

  2004cited by this paper
• Statistical Phrase-Based Translation

  2003cited by this paper
• A Systematic Comparison of Various Statistical Alignment Models

  2003cited by this paper
• Bleu: a Method for Automatic Evaluation of Machine Translation

  2002cited by this paper
• Stochastic Inversion Transduction Grammars and Bilingual Parsing of Parallel Corpora

  1997cited by this paper
• The Mathematics of Statistical Machine Translation: Parameter Estimation

  1993cited by this paper
• An Indoeuropean classification : a lexicostatistical experiment

  1992cited by this paper
• Lexico-Statistical Dating of Prehistoric Ethnic Contacts

  1952cited by this paper

• Assessing the Impact of Typological Features on Multilingual Machine Translation in the Age of Large Language Models

  2026cites this paper
• A Cross-Lingual Perspective on Neural Machine Translation Difficulty

  2025cites this paper
• Zero-Shot Performance Prediction for Probabilistic Scaling Laws

  2025cites this paper
• Pivot Language for Low-Resource Machine Translation

  2025cites this paper
• Limitations of Religious Data and the Importance of the Target Domain: Towards Machine Translation for Guinea-Bissau Creole

  2025cites this paper
• Modeling of learning curves with applications to POS tagging

  2024cites this paper
• Hybrid Preferences: Learning to Route Instances for Human vs. AI Feedback

  2024cites this paper
• ProxyLM: Predicting Language Model Performance on Multilingual Tasks via Proxy Models

  2024cites this paper
• Predicting Machine Translation Performance on Low-Resource Languages: The Role of Domain Similarity

  2024cites this paper
• Multi Task Learning For Zero Shot Performance Prediction of Multilingual Models

  2022influential citation
• Detecting Languages Unintelligible to Multilingual Models through Local Structure Probes

  2022cites this paper
• Surfing the modeling of PoS taggers in low-resource scenarios

  2022cites this paper
• Learning from flowsheets: A generative transformer model for autocompletion of flowsheets

  2022influential citation
• DivEMT: Neural Machine Translation Post-Editing Effort Across Typologically Diverse Languages

  2022cites this paper
• Consistent Human Evaluation of Machine Translation across Language Pairs

  2022cites this paper
• Are All the Datasets in Benchmark Necessary? A Pilot Study of Dataset Evaluation for Text Classification

  2022influential citation
• On Systematic Style Differences between Unsupervised and Supervised MT and an Application for High-Resource Machine Translation

  2021cites this paper
• Neighbourhood-based Cross-Language Ontology Matching

  2021cites this paper
• A Large-Scale Study of Machine Translation in Turkic Languages

  2021cites this paper
• On the Difficulty of Translating Free-Order Case-Marking Languages

  2021cites this paper
• What Can Unsupervised Machine Translation Contribute to High-Resource Language Pairs?

  2021cites this paper
• Family of Origin and Family of Choice: Massively Parallel Lexiconized Iterative Pretraining for Severely Low Resource Text-based Translation

  2021cites this paper
• Family of Origin and Family of Choice: Massively Parallel Lexiconized Iterative Pretraining for Severely Low Resource Machine Translation

  2021cites this paper
• Towards More Fine-grained and Reliable NLP Performance Prediction

  2021cites this paper
• Domain Divergences: A Survey and Empirical Analysis

  2020influential citation
• MEDLINE as a Parallel Corpus: a Survey to Gain Insight on French-, Spanish- and Portuguese-speaking Authors’ Abstract Writing Practice

  2020cites this paper
• Fine-Grained Analysis of Cross-Linguistic Syntactic Divergences

  2020cites this paper
• It’s Easier to Translate out of English than into it: Measuring Neural Translation Difficulty by Cross-Mutual Information

  2020cites this paper
• Predicting Performance for Natural Language Processing Tasks

  2020cites this paper
• Predicting syntactic equivalence between source and target sentences

  2019influential citation
• Machine Translation from Natural Language to Code using Long-Short Term Memory

  2019cites this paper
• On the Relation between Linguistic Typology and (Limitations of) Multilingual Language Modeling

  2018cites this paper
• Monotonic Chunkwise Attention

  2017cites this paper
• The mat sat on the cat: investigating structure in the evaluation of order in machine translation

  2017cites this paper
• A Parallel Corpus for Evaluating Machine Translation between Arabic and European Languages

  2017cites this paper
• Investigating the Relationship between Classification Quality and SMT Performance in Discriminative Reordering Models

  2017cites this paper
• DTED: Evaluation of Machine Translation Structure Using Dependency Parsing and Tree Edit Distance

  2016cites this paper
• Modèles exponentiels et contraintes sur les espaces de recherche en traduction automatique et pour le transfert cross-lingue. (Log-linear Models and Search Space Constraints in Statistical Machine Translation and Cross-lingual Transfer)

  2016cites this paper
• Reordering space design in statistical machine translation

  2016cites this paper
• Prior estimation of post accuracy

  2016cites this paper
• Measuring acceptability of machine translated enterprise content

  2016cites this paper
• Hierarchical Permutation Complexity for Word Order Evaluation

  2016cites this paper
• Linguistic Structure in Statistical Machine Translation

  2015cites this paper
• Source discriminative word lexicon for translation disambiguation

  2015cites this paper
• Large Scale Translation Quality Estimation

  2015cites this paper
• Surveys: A Survey of Word Reordering in Statistical Machine Translation: Computational Models and Language Phenomena

  2015cites this paper
• Improving statistical machine translation using syntax-based learning-to-rank system

  2015cites this paper
• Non-projective Dependency-based Pre-Reordering with Recurrent Neural Network for Machine Translation

  2015cites this paper
• Multi-lingual Geoparsing based on Machine Translation

  2015cites this paper
• Non-projective Dependency-based Pre-Reordering with Recurrent Neural Network for Machine Translation

  2015cites this paper
• Rule-based Reordering Space in Statistical Machine Translation

  2014cites this paper
• Pre-ordering of phrase-based machine translation input in translation workflow

  2014cites this paper
• Understanding the Performance of Statistical MT Systems : A Linear Regression Framework F rancisco

  2013cites this paper
• An information-theoretic measure to evaluate parsing difficulty across treebanks

  2013cites this paper
• How to Choose the Best Pivot Language for Automatic Translation of Low-Resource Languages

  2013cites this paper
• The Critical Technology Development Status of Machine Translation

  2013cites this paper
• Linguistically Motivated Reordering Modeling for Phrase-Based Statistical Machine Translation

  2013influential citation
• Generation of Compound Words in Statistical Machine Translation into Compounding Languages

  2013cites this paper
• Predicting sentence translation quality using extrinsic and language independent features

  2013cites this paper
• Efficient Solutions for Word Reordering in German-English Phrase-Based Statistical Machine Translation

  2013cites this paper
• Pre-Reordering for Machine Translation Using Transition-Based Walks on Dependency Parse Trees

  2013cites this paper
• Text Harmonization Strategies for Phrase-Based Statistical Machine Translation

  2012cites this paper
• Predicting alignment performance

  2012cites this paper
• How Good are Typological Distances for Determining Genealogical Relationships among Languages?

  2012cites this paper
• Reordering in statistical machine translation

  2012cites this paper
• Understanding the Performance of Statistical MT Systems: A Linear Regression Framework

  2012cites this paper
• Prediction of Learning Curves in Machine Translation

  2012cites this paper
• Evaluating Indirect Strategies for Chinese - Spanish Statistical Machine Translation: Extended Abstract

  2012influential citation
• Reordering metrics for statistical machine translation

  2011cites this paper
• An exponential translation model for target language morphology

  2011cites this paper
• Impact of corpus size and quality on English-Bangla statistical Machine Translation system

  2011cites this paper
• Pre- and Postprocessing for Statistical Machine Translation into Germanic Languages

  2011cites this paper
• Soft Dependency Constraints for Reordering in Hierarchical Phrase-Based Translation

  2011cites this paper
• A Lightweight Evaluation Framework for Machine Translation Reordering

  2011cites this paper
• Translation Quality Indicators for Pivot-based Statistical MT

  2011cites this paper
• La traduction automatique des pronoms. Problèmes et perspectives (Automatic translation of pronouns. Problems and perspectives)

  2011cites this paper
• Metrics for MT evaluation: evaluating reordering

  2010influential citation
• Multilingual Pseudo-Relevance Feedback: Performance Study of Assisting Languages

  2010cites this paper
• Structured local exponential models for machine translation

  2010cites this paper
• Compound Processing for Phrase-Based Statistical Machine Translation

  2009cites this paper
• Performance Confidence Estimation for Automatic Summarization

  2009cites this paper
• On the Importance of Pivot Language Selection for Statistical Machine Translation

  2009cites this paper
• A Quantitative Analysis of Reordering Phenomena

  2009influential citation
• Word Lattices for Multi-Source Translation

  2009cites this paper
• 462 Machine Translation Systems for Europe

  2009influential citation
• Decoding by Dynamic Chunking for Statistical Machine Translation

  2009cites this paper
• Conundrums in Noun Phrase Coreference Resolution: Making Sense of the State-of-the-Art

  2009cites this paper
• A unified framework for phrase-based, hierarchical, and syntax-based statistical machine translation

  2009cites this paper