Manual and Automatic Evaluation of Machine Translation between European Languages

We evaluated machine translation performance for six European language pairs that participated in a shared task: translating French, German, Spanish texts to English and back. Evaluation was done automatically using the Bleu score and manually on fluency and adequacy.

• Publication year

  2006

• Venue

  WMT@HLT-NAACL

• Publication date

  2006-06-08

• Fields of study

  Linguistics, Computer Science

• Identifiers
  DOI 10.3115/1654650.1654666
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Re-evaluating the Role of Bleu in Machine Translation Research

  2006cited by this paper
• Syntax Augmented Machine Translation via Chart Parsing

  2006cited by this paper
• Stochastic Inversion Transduction Grammars for Obtaining Word Phrases for Phrase-based Statistical Machine Translation

  2006cited by this paper
• Mood at work: Ramses versus Pharaoh

  2006cited by this paper
• N-gram-based SMT System Enhanced with Reordering Patterns

  2006cited by this paper
• Constraining the Phrase-Based, Joint Probability Statistical Translation Model

  2006influential reference
• NTT System Description for the WMT2006 Shared Task

  2006cited by this paper
• Microsoft Research Treelet Translation System: NAACL 2006 Europarl Evaluation

  2006cited by this paper
• Phramer - An Open Source Statistical Phrase-Based Translator

  2006cited by this paper
• TALP Phrase-based statistical translation system for European language pairs

  2006cited by this paper
• The LDV-COMBO system for SMT

  2006cited by this paper
• PORTAGE: with Smoothed Phrase Tables and Segment Choice Models

  2006cited by this paper
• Language Models and Reranking for Machine Translation

  2006cited by this paper
• Overview of the IWSLT 2005 Evaluation Campaign

  2005cited by this paper
• Shared Task: Statistical Machine Translation between European Languages

  2005cited by this paper
• On Some Pitfalls in Automatic Evaluation and Significance Testing for MT

  2005cited by this paper
• Clause Restructuring for Statistical Machine Translation

  2005cited by this paper
• Statistical Significance Tests for Machine Translation Evaluation

  2004cited by this paper

• Revisiting Metric Reliability for Fine-grained Evaluation of Machine Translation and Summarization in Indian Languages

  2025cites this paper
• Metamorphic Relation Patterns for Metamorphic Testing, Exploration and Robustness

  2025cites this paper
• When the Gold Standard isn't Necessarily Standard: Challenges of Evaluating the Translation of User-Generated Content

  2025cites this paper
• Limitations of Religious Data and the Importance of the Target Domain: Towards Machine Translation for Guinea-Bissau Creole

  2025cites this paper
• Comparative analysis of machine translation for Hindi-Dogri text using rule-based, statistical, and neural approaches

  2025cites this paper
• Self-Vocabularizing Training for Neural Machine Translation

  2025cites this paper
• Refined Assessment for Translation Evaluation: Rethinking Machine Translation Evaluation in the Era of Human-Level Systems

  2025cites this paper
• Teaching Language Models To Gather Information Proactively

  2025influential citation
• Findings of the AmericasNLP 2025 Shared Tasks on Machine Translation, Creation of Educational Material, and Translation Metrics for Indigenous Languages of the Americas

  2025cites this paper
• Audio-Based Crowd-Sourced Evaluation of Machine Translation Quality

  2025cites this paper
• Advancing Bangla Machine Translation Through Informal Datasets

  2025cites this paper
• An Interdisciplinary Approach to Human-Centered Machine Translation

  2025cites this paper
• Exploring GPT-4 Capabilities in Generating Paraphrased Sentences for the Arabic Language

  2025cites this paper
• Arabizi vs LLMs: Can the Genie Understand the Language of Aladdin?

  2025influential citation
• You Cannot Feed Two Birds with One Score: the Accuracy-Naturalness Tradeoff in Translation

  2025cites this paper
• Enhancing Human Evaluation in Machine Translation with Comparative Judgment

  2025cites this paper
• Guardians of the Machine Translation Meta-Evaluation: Sentinel Metrics Fall In!

  2024cites this paper
• Evaluation Briefs: Drawing on Translation Studies for Human Evaluation of MT

  2024cites this paper
• The link between translation difficulty and the quality of machine translation: a literature review and empirical investigation

  2024cites this paper
• A Linguistic Investigation for a Case Study of ChatGPT and Google Translate in Rendering Special Needs Texts from English into Arabic: A Synchronic Case Study

  2024cites this paper
• Pitfalls and Outlooks in Using COMET

  2024cites this paper
• Improving Statistical Significance in Human Evaluation of Automatic Metrics via Soft Pairwise Accuracy

  2024cites this paper
• Error Span Annotation: A Balanced Approach for Human Evaluation of Machine Translation

  2024cites this paper
• Calibration and context in human evaluation of machine translation

  2024cites this paper
• Comparison of Evaluation Metrics for Short Story Generation

  2023cites this paper
• Multifaceted Challenge Set for Evaluating Machine Translation Performance

  2023cites this paper
• Data Sampling and (In)stability in Machine Translation Evaluation

  2023cites this paper
• Cognitive load and cognitive effort

  2023cites this paper
• Quality in Human and Machine Translation: An Interdisciplinary Survey

  2023cites this paper
• The Role of Compounds in Human vs. Machine Translation Quality

  2023cites this paper
• Word Closure-Based Metamorphic Testing for Machine Translation

  2023cites this paper
• High-Resource Methodological Bias in Low-Resource Investigations

  2022cites this paper
• Investigating the Impact of Different Pivot Languages on Translation Quality

  2022cites this paper
• NarraSum: A Large-Scale Dataset for Abstractive Narrative Summarization

  2022cites this paper
• Are Pretrained Multilingual Models Equally Fair across Languages?

  2022cites this paper
• Consistent Human Evaluation of Machine Translation across Language Pairs

  2022cites this paper
• XInfoTabS: Evaluating Multilingual Tabular Natural Language Inference

  2022cites this paper
• PROPOSITION-BASED EVALUATION OF MACHINE-TRANSLATED ACADEMIC TEXT

  2022cites this paper
• An Overview on Machine Translation Evaluation

  2022influential citation
• How Effective is Byte Pair Encoding for Out-Of-Vocabulary Words in Neural Machine Translation?

  2022cites this paper
• Extrinsic Evaluation of Machine Translation Metrics

  2022cites this paper
• Cloze Evaluation for Deeper Understanding of Commonsense Stories in Indonesian

  2022cites this paper
• Read Top News First: A Document Reordering Approach for Multi-Document News Summarization

  2022cites this paper
• Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text

  2022cites this paper
• Empirical Evaluation of Language Agnostic Filtering of Parallel Data for Low Resource Languages

  2022cites this paper
• Translation Quality Assessment: A Brief Survey on Manual and Automatic Methods

  2021influential citation
• Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation

  2021cites this paper
• Human evaluation of three machine translation systems: from quality to attitudes by professional translators

  2021cites this paper
• Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort

  2021cites this paper
• Grammar Accuracy Evaluation (GAE): Quantifiable Intrinsic Evaluation of Machine Translation Models

  2021cites this paper
• The REPU CS’ Spanish–Quechua Submission to the AmericasNLP 2021 Shared Task on Open Machine Translation

  2021cites this paper
• Findings of the 2021 Conference on Machine Translation (WMT21)

  2021cites this paper
• Grammar Accuracy Evaluation (GAE): Quantifiable Qualitative Evaluation of Machine Translation Models

  2021cites this paper
• Goal-Driven Visual Question Generation from Radiology Images

  2021cites this paper
• Out of the BLEU: An Error Analysis of Statistical and Neural Machine Translation of WikiHow Articles from English into Arabic

  2021cites this paper
• Survey of Low-Resource Machine Translation

  2021cites this paper
• On nature and causes of observed MT errors

  2021cites this paper
• An Evaluation of the Accuracy of the Machine Translation Systems of Social Media Language

  2021cites this paper
• Towards Computational Linguistics in Minangkabau Language: Studies on Sentiment Analysis and Machine Translation

  2020cites this paper
• Machine Translation System for Translation of Malayalam Morphological Causative Constructions into English Periphrastic Causative

  2020cites this paper
• The ergonomics of translation tools: understanding when less is actually more

  2020cites this paper
• Findings of the 2020 Conference on Machine Translation (WMT20)

  2020cites this paper
• SemMT: A Semantic-Based Testing Approach for Machine Translation Systems

  2020cites this paper
• Machine translation models for Cantonese-English translation

  2020cites this paper
• Sentence Level Human Translation Quality Estimation with Attention-based Neural Networks

  2020cites this paper
• Informative Manual Evaluation of Machine Translation Output

  2020cites this paper
• SemMT: A Semantic-based Testing Approach for Machine Translation Systems

  2020cites this paper
• Statistical Machine Translation Customization between Turkish and 11 Languages

  2020cites this paper
• Visual Question Generation from Radiology Images

  2020cites this paper
• Relations between comprehensibility and adequacy errors in machine translation output

  2020cites this paper
• NICE: Neural Integrated Custom Engines

  2020cites this paper
• What’s the Difference Between Professional Human and Machine Translation? A Blind Multi-language Study on Domain-specific MT

  2020cites this paper
• Adequacy in Machine vs. Human Translation: A Comparative Study of English and Persian Languages

  2020cites this paper
• FFR v1.1: Fon-French Neural Machine Translation

  2020cites this paper
• Bridging the Structural Gap Between Encoding and Decoding for Data-To-Text Generation

  2020cites this paper
• QUES: A Quality Estimation System of Arabic to English Translation

  2020cites this paper
• BLEU Might Be Guilty but References Are Not Innocent

  2020cites this paper
• Toward General Scene Graph: Integration of Visual Semantic Knowledge with Entity Synset Alignment

  2020cites this paper
• Enabling Robust Grammatical Error Correction in New Domains: Data Sets, Metrics, and Analyses

  2019cites this paper
• How to evaluate machine translation: A review of automated and human metrics

  2019cites this paper
• Results of the WMT19 Metrics Shared Task: Segment-Level and Strong MT Systems Pose Big Challenges

  2019cites this paper
• Taking MT Evaluation Metrics to Extremes: Beyond Correlation with Human Judgments

  2019cites this paper
• Evaluating Rewards for Question Generation Models

  2019cites this paper
• HEAD-QA: A Healthcare Dataset for Complex Reasoning

  2019cites this paper
• Spanish-Swedish neural machine translation for the civil engineering domain

  2019cites this paper
• English-Bhojpuri SMT System: Insights from the Karaka Model

  2019cites this paper
• English-Bhojpuri SMT System: Insights from the Karaka Model

  2019cites this paper
• Helping the helpers: Evaluating the impact of a controlled language checker on the intralingual and interlingual translation tasks involving volunteer health professionals

  2019cites this paper
• Simplifying, reading, and machine translating health content: an empirical investigation of usability

  2019cites this paper
• Findings of the 2019 Conference on Machine Translation (WMT19)

  2019cites this paper
• Findings of the 2018 Conference on Machine Translation (WMT18)

  2018cites this paper
• Results of the WMT18 Metrics Shared Task: Both characters and embeddings achieve good performance

  2018cites this paper
• English-to-Czech MT: Large Data and Beyond

  2018cites this paper
• BLEU is Not Suitable for the Evaluation of Text Simplification

  2018cites this paper
• Enhancement of post-editing performance: introducing machine translation post-editing in translator training

  2018cites this paper
• A Monte Carlo Method for Metamorphic Testing of Machine Translation Services

  2018cites this paper
• Metamorphic Testing for Machine Translations: MT4MT

  2018cites this paper
• English-Basque Statistical and Neural Machine Translation

  2018cites this paper
• Preference learning for machine translation

  2018cites this paper
• No Longer Lost in Translation: Evidence that Google Translate Works for Comparative Bag-of-Words Text Applications

  2018cites this paper