Discriminative Corpus Weight Estimation for Machine Translation

Current statistical machine translation (SMT) systems are trained on sentence-aligned and word-aligned parallel text collected from various sources. Translation model parameters are estimated from the word alignments, and the quality of the translations on a given test set depends on the parameter estimates. There are at least two factors affecting the parameter estimation: domain match and training data quality. This paper describes a novel approach for automatically detecting and down-weighing certain parts of the training corpus by assigning a weight to each sentence in the training bitext so as to optimize a discriminative objective function on a designated tuning set. This way, the proposed method can limit the negative effects of low quality training data, and can adapt the translation model to the domain of interest. It is shown that such discriminative corpus weights can provide significant improvements in Arabic-English translation on various conditions, using a state-of-the-art SMT system.

• Publication year

  2009

• Venue

  Conference on Empirical Methods in Natural Language Processing

• Publication date

  2009-08-06

• Fields of study

  Computer Science

• Identifiers
  DOI 10.3115/1699571.1699605
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Language and Translation Model Adaptation using Comparable Corpora

  2008cited by this paper
• A New String-to-Dependency Machine Translation Algorithm with a Target Dependency Language Model

  2008cited by this paper
• Efficient data selection for machine translation

  2008cited by this paper
• Method of Selecting Training Data to Build a Compact and Efficient Translation Model

  2008cited by this paper
• Improving Statistical Machine Translation Performance by Training Data Selection and Optimization

  2007cited by this paper
• METEOR: An Automatic Metric for MT Evaluation with High Levels of Correlation with Human Judgments

  2007influential reference
• A Study of Translation Edit Rate with Targeted Human Annotation

  2006cited by this paper
• A STUDY OF TRANSLATION ERROR RATE WITH TARGETED HUMAN ANNOTATION

  2005cited by this paper
• Adaptation of the translation model for statistical machine translation based on information retrieval

  2005cited by this paper
• Pharaoh: a beam search decoder for phrase-based statistical machine translation models

  2004cited by this paper
• Pharaoh: A Beam Search Decoder for Phrase-Based Statistical Machine Translation Models

  2004cited by this paper
• The Web as a Parallel Corpus

  2003cited by this paper
• Minimum Error Rate Training in Statistical Machine Translation

  2003influential reference
• A Systematic Comparison of Various Statistical Alignment Models

  2003influential reference
• Bleu: a Method for Automatic Evaluation of Machine Translation

  2002influential reference
• Integration of Diverse Recognition Methodologies Through Reevaluation of N-Best Sentence Hypotheses

  1991cited by this paper
• On the limited memory BFGS method for large scale optimization

  1989cited by this paper
• Algorithms for minimization without derivatives

  1974cited by this paper
• An efficient method for finding the minimum of a function of several variables without calculating derivatives

  1964cited by this paper
• Edinburgh Research Explorer Experiments in Domain Adaptation for Statistical Machine Translation

  year unknowncited by this paper

• Improved Unsupervised Statistical Machine Translation via Unsupervised Word Sense Disambiguation for a Low-Resource and Indic Languages

  2022cites this paper
• A Survey of Domain Adaptation for Machine Translation

  2020cites this paper
• Curriculum Learning for Domain Adaptation in Neural Machine Translation

  2019cites this paper
• Word-based Domain Adaptation for Neural Machine Translation

  2019cites this paper
• Denoising Neural Machine Translation Training with Trusted Data and Online Data Selection

  2018cites this paper
• Mixing Textual Data Selection Methods for Improved In-Domain Data Adaptation

  2018cites this paper
• Detecting Machine-translated Subtitles in Large Parallel Corpora

  2018cites this paper
• Domain Adaptation for Statistical Machine Translation

  2018cites this paper
• A Re-Ranker Scheme For Integrating Large Scale NLU Models

  2018cites this paper
• Freezing Subnetworks to Analyze Domain Adaptation in Neural Machine Translation

  2018cites this paper
• Evaluation of Machine Translation Performance Across Multiple Genres and Languages

  2018cites this paper
• Document-Level Information as Side Constraints for Improved Neural Patent Translation

  2018cites this paper
• Models for Translation Domain Adaptation

  2018cites this paper
• Document-Level Information as Side Constraints for Improved Neural Patent Translation

  2018cites this paper
• - 1101 Data Selection using Topic Adaptation for Statistical Machine Translation

  2018cites this paper
• A Survey of Domain Adaptation for Neural Machine Translation

  2018cites this paper
• Sentence Selection and Weighting for Neural Machine Translation Domain Adaptation

  2018cites this paper
• Domain adaptation using neural network joint model

  2017cites this paper
• A survey of domain adaptation for statistical machine translation

  2017cites this paper
• Fine-Tuning for Neural Machine Translation with Limited Degradation across In- and Out-of-Domain Data

  2017cites this paper
• A simplification-translation-restoration framework for domain adaptation in statistical machine translation: A case study in medical record translation

  2017cites this paper
• Instance Weighting for Neural Machine Translation Domain Adaptation

  2017influential citation
• Exploiting Relative Frequencies for Data Selection

  2017cites this paper
• Domain adaptation for statistical machine translation

  2017cites this paper
• Applications of Topic Models

  2017cites this paper
• Cost Weighting for Neural Machine Translation Domain Adaptation

  2017cites this paper
• What ’ s in a Domain ? Analyzing Genre and Topic Differences in SMT

  2017cites this paper
• Latent domain models for statistical machine translation

  2017cites this paper
• Domain adaptation for statistical machine translation

  2016cites this paper
• Semi-supervised Convolutional Networks for Translation Adaptation with Tiny Amount of In-domain Data

  2016cites this paper
• A Loss-Augmented Approach to Training Syntactic Machine Translation Systems

  2016cites this paper
• A Deep Fusion Model for Domain Adaptation in Phrase-based MT

  2016cites this paper
• Fast Domain Adaptation for Neural Machine Translation

  2016cites this paper
• Multi-domain Adaptation for Statistical Machine Translation Based on Feature Augmentation

  2016cites this paper
• Adapting to All Domains at Once: Rewarding Domain Invariance in SMT

  2016cites this paper
• Research on Domain Adaptation for SMT Based on Specific Domain Knowledge

  2016cites this paper
• JU-USAAR: A Domain Adaptive MT System

  2016cites this paper
• Topic Model Based Adaptation Data Selection for Domain-Specific Machine Translation

  2016cites this paper
• Mixed domain vs. multi-domain statistical machine translation

  2015cites this paper
• How to Avoid Unwanted Pregnancies: Domain Adaptation using Neural Network Models

  2015cites this paper
• Data Selection using Topic Adaptation for Statistical Machine Translation

  2015cites this paper
• Using joint models or domain adaptation in statistical machine translation

  2015cites this paper
• What’s in a Domain? Analyzing Genre and Topic Differences in Statistical Machine Translation

  2015cites this paper
• Translation Model Adaptation Using Genre-Revealing Text Features

  2015cites this paper
• Improving translation quality stability using Bayesian predictive adaptation

  2015cites this paper
• Resampling approach for instance-based domain adaptation from patent domain to newspaper domain in statistical machine translation

  2015cites this paper
• Domain Adaptation for SMT Using Sentence Weight

  2015cites this paper
• Five Shades of Noise: Analyzing Machine Translation Errors in User-Generated Text

  2015cites this paper
• Refinements in hierarchical phrase-based translation systems

  2015cites this paper
• Domain-Specific Paraphrase Extraction

  2015cites this paper
• Discriminative Reordering Model Adaptation via Structural Learning

  2015cites this paper
• A novel method to optimize training data for translation model adaptation

  2015cites this paper
• Dynamic Topic Adaptation for Improved Contextual Modelling in Statistical Machine Translation

  2015cites this paper
• Dynamic Topic Adaptation for Phrase-based MT

  2014cites this paper
• Anticipatory translation model adaptation for bilingual conversations

  2014cites this paper
• Machine translation for e-government – the Baltic case

  2014cites this paper
• Latent Domain Translation Models in Mix-of-Domains Haystack

  2014cites this paper
• Comparison of data selection techniques for the translation of video lectures

  2014cites this paper
• Translation model based weighting for phrase extraction

  2014cites this paper
• Latent Domain Phrase-based Models for Adaptation

  2014cites this paper
• Translation project adaptation for MT-enhanced computer assisted translation

  2014cites this paper
• A comparison of mixture and vector space techniques for translation model adaptation

  2014cites this paper
• Adaptive HTER Estimation for Document-Specific MT Post-Editing

  2014cites this paper
• Combining domain and topic adaptation for SMT

  2014cites this paper
• A Systematic Comparison of Data Selection Criteria for SMT Domain Adaptation

  2014cites this paper
• On-demand Development of Statistical Machine Translation Systems. (Développement à la demande des systèmes de traduction automatique statistiques)

  2014cites this paper
• Models, Inference, and Implementation for Scalable Probabilistic Models of Text

  2014cites this paper
• Adaptation in Machine Translation

  2014cites this paper
• THE EFFECT OF PARALLEL CORPUS QUALITY VS SIZE IN ENGLISH -TO- TURKISH SMT

  2014cites this paper
• UvA-DARE (Digital Academic Repository) Latent Domain Phrase-based Models for Adaptation

  2014cites this paper
• Polylingual Tree-Based Topic Models for Translation Domain Adaptation

  2014cites this paper
• Improving MT post-editing productivity with adaptive confidence estimation for document-specific translation model

  2014cites this paper
• iCPE: A Hybrid Data Selection Model for SMT Domain Adaptation

  2013cites this paper
• Incremental Topic-Based Translation Model Adaptation for Conversational Spoken Language Translation

  2013cites this paper
• Parameter Optimization for Statistical Machine Translation: It Pays to Learn from Hard Examples

  2013cites this paper
• Improved Online Learning and Modeling for Feature-Rich Discriminative Machine Translation

  2013cites this paper
• Topic Models for Translation Domain Adaptation

  2013cites this paper
• Uses of Monolingual In-Domain Corpora for Cross-Domain Adaptation with Hybrid MT Approaches

  2013cites this paper
• Vector Space Model for Adaptation in Statistical Machine Translation

  2013cites this paper
• Phrase Training Based Adaptation for Statistical Machine Translation

  2013cites this paper
• Measuring Machine Translation Errors in New Domains

  2013cites this paper
• Collective Corpus Weighting and Phrase Scoring for SMT Using Graph-Based Random Walk

  2013cites this paper
• Project Adaptation for MT-Enhanced Computer Assisted Translation

  2013cites this paper
• A Multi-Domain Translation Model Framework for Statistical Machine Translation

  2013cites this paper
• Domain adaptation for translation models in statistical machine translation

  2013cites this paper
• Bagging and Boosting statistical machine translation systems

  2013cites this paper
• Issues in incremental adaptation of statistical MT from human post-edits

  2013cites this paper
• Adaptation of Reordering Models for Statistical Machine Translation

  2013cites this paper
• LEVERAGING DIVERSE SOURCES IN STATISTICAL MACHINE TRANSLATION

  2013cites this paper
• What is Hidden among Translation Rules

  2013cites this paper
• The (Un)faithful Machine Translator

  2013cites this paper
• Perplexity Minimization for Translation Model Domain Adaptation in Statistical Machine Translation

  2012cites this paper
• Topic Models for Dynamic Translation Model Adaptation

  2012cites this paper
• Towards contextual adaptation for any-text translation

  2012cites this paper
• A General Framework to Weight Heterogeneous Parallel Data for Model Adaptation in Statistical MT

  2012cites this paper
• Does more data always yield better translations?

  2012cites this paper
• Incremental Re-Training of a Hybrid English-French MT System with Customer Translation Memory Data

  2012cites this paper
• Automatic Tune Set Generation for Machine Translation with Limited Indomain Data

  2012influential citation
• Applications of data selection via cross-entropy difference for real-world statistical machine translation

  2012cites this paper
• A simple and effective weighted phrase extraction for machine translation adaptation

  2012cites this paper