Quality Estimation and Translation Metrics via Pre-trained Word and Sentence Embeddings

We propose the use of pre-trained embeddings as features of a regression model for sentence-level quality estimation of machine translation. In our work we combine freely available BERT and LASER multilingual embeddings to train a neural-based regression model. In the second proposed method we use as an input features not only pre-trained embeddings, but also log probability of any machine translation (MT) system. Both methods are applied to several language pairs and are evaluated both as a classical quality estimation system (predicting the HTER score) as well as an MT metric (predicting human judgements of translation quality).

• Publication year

  2019

• Venue

  Conference on Machine Translation

• Publication date

  2019-08-01

• Fields of study

  Linguistics, Computer Science

• Identifiers
  DOI 10.18653/v1/W19-5410
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Findings of the WMT 2019 Shared Tasks on Quality Estimation

  2019cited by this paper
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

  2019influential reference
• "Bilingual Expert" Can Find Translation Errors

  2018cited by this paper
• Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond

  2018influential reference
• Findings of the 2018 Conference on Machine Translation (WMT18)

  2018cited by this paper
• Findings of the WMT 2018 Shared Task on Quality Estimation

  2018cited by this paper
• Results of the WMT18 Metrics Shared Task: Both characters and embeddings achieve good performance

  2018cited by this paper
• deepQuest: A Framework for Neural-based Quality Estimation

  2018cited by this paper
• Results of the WMT17 Metrics Shared Task

  2017cited by this paper
• Predictor-Estimator using Multilevel Task Learning with Stack Propagation for Neural Quality Estimation

  2017cited by this paper
• chrF++: words helping character n-grams

  2017cited by this paper
• Attention is All you Need

  2017influential reference
• Findings of the 2017 Conference on Machine Translation (WMT17)

  2017cited by this paper
• Findings of the 2016 Conference on Machine Translation

  2016influential reference
• Results of the WMT16 Metrics Shared Task

  2016influential reference
• Findings of the 2015 Workshop on Statistical Machine Translation

  2015cited by this paper
• Can machine translation systems be evaluated by the crowd alone

  2015cited by this paper
• Results of the WMT15 Metrics Shared Task

  2015cited by this paper
• Adam: A Method for Stochastic Optimization

  2014cited by this paper
• Estimating the Sentence-Level Quality of Machine Translation Systems

  2009cited by this paper
• A Study of Translation Edit Rate with Targeted Human Annotation

  2006influential reference
• A STUDY OF TRANSLATION ERROR RATE WITH TARGETED HUMAN ANNOTATION

  2005cited by this paper
• Confidence Estimation for Machine Translation

  2004cited by this paper
• Bleu: a Method for Automatic Evaluation of Machine Translation

  2002cited by this paper

• From Handcrafted Features to LLMs: A Brief Survey for Machine Translation Quality Estimation

  2024cites this paper
• Towards Precise Localization of Critical Errors in Machine Translation

  2024cites this paper
• Linguistic Communication Channels Reveal Connections between Texts: The New Testament and Greek Literature

  2023cites this paper
• A Holistic Approach to Reference-Free Evaluation of Machine Translation

  2023cites this paper
• A New Approach to Quality Assessment of Chinese-English Neural Machine Translation

  2023cites this paper
• Toxicity in Multilingual Machine Translation at Scale

  2022cites this paper
• KG-BERTScore: Incorporating Knowledge Graph into BERTScore for Reference-Free Machine Translation Evaluation

  2022cites this paper
• Target-Side Language Model for Reference-Free Machine Translation Evaluation

  2022cites this paper
• Linguistically Driven Multi-Task Pre-Training for Low-Resource Neural Machine Translation

  2022cites this paper
• ELIZAVETA YANKOVSKAYA Quality Estimation through Attention

  2022cites this paper
• Better Quality Estimation for Low Resource Corpus Mining

  2022cites this paper
• Exploiting Curriculum Learning in Unsupervised Neural Machine Translation

  2021cites this paper
• Assessing Reference-Free Peer Evaluation for Machine Translation

  2021cites this paper
• CLIPScore: A Reference-free Evaluation Metric for Image Captioning

  2021cites this paper
• Variance-Aware Machine Translation Test Sets

  2021cites this paper
• Evolution of Gaussian Process kernels for machine translation post-editing effort estimation

  2021cites this paper
• An Oblivious Approach to Machine Translation Quality Estimation

  2021cites this paper
• Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasing

  2020cites this paper
• The NiuTrans System for the WMT20 Quality Estimation Shared Task

  2020cites this paper
• Translation Quality Estimation by Jointly Learning to Score and Rank

  2020cites this paper
• Zero-Shot Translation Quality Estimation with Explicit Cross-Lingual Patterns

  2020cites this paper
• On the Limitations of Cross-lingual Encoders as Exposed by Reference-Free Machine Translation Evaluation

  2020influential citation
• Selecting Backtranslated Data from Multiple Sources for Improved Neural Machine Translation

  2020cites this paper
• Results of the WMT19 Metrics Shared Task: Segment-Level and Strong MT Systems Pose Big Challenges

  2019cites this paper
• Findings of the WMT 2019 Shared Tasks on Quality Estimation

  2019cites this paper