Findings of the Second Shared Task on Multimodal Machine Translation and Multilingual Image Description

We present the results from the second shared task on multimodal machine translation and multilingual image description. Nine teams submitted 19 systems to two tasks. The multimodal translation task, in which the source sentence is supplemented by an image, was extended with a new language (French) and two new test sets. The multilingual image description task was changed such that at test time, only the image is given. Compared to last year, multimodal systems improved, but text-only systems remain competitive.

• Publication year

  2017

• Venue

  Conference on Machine Translation

• Publication date

  2017-10-01

• Fields of study

  Linguistics, Computer Science

• Identifiers
  DOI 10.18653/v1/W17-4718arXiv 1710.07177
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measures (Extended Abstract)

  2017cited by this paper
• CUNI System for the WMT17 Multimodal Translation Task

  2017cited by this paper
• The AFRL-OSU WMT17 Multimodal Translation System: An Image Processing Approach

  2017cited by this paper
• NICT-NAIST System for WMT17 Multimodal Translation Task

  2017influential reference
• Nematus: a Toolkit for Neural Machine Translation

  2017cited by this paper
• Sheffield MultiMT: Using Object Posterior Predictions for Multimodal Machine Translation

  2017cited by this paper
• Generating Image Descriptions using Multilingual Data

  2017cited by this paper
• Attention Strategies for Multi-Source Sequence-to-Sequence Learning

  2017cited by this paper
• Imagination Improves Multimodal Translation

  2017cited by this paper
• STAIR Captions: Constructing a Large-Scale Japanese Image Caption Dataset

  2017cited by this paper
• Incorporating Global Visual Features into Attention-based Neural Machine Translation.

  2017cited by this paper
• DCU System Report on the WMT 2017 Multi-modal Machine Translation Task

  2017influential reference
• Cross-linguistic differences and similarities in image descriptions

  2017cited by this paper
• NMTPY: A Flexible Toolkit for Advanced Neural Machine Translation Systems

  2017cited by this paper
• LIUM-CVC Submissions for WMT17 Multimodal Translation Task

  2017cited by this paper
• OSU Multimodal Machine Translation System Report

  2017cited by this paper
• Cross-Lingual Image Caption Generation

  2016cited by this paper
• Adding Chinese Captions to Images

  2016cited by this paper
• Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measures

  2016cited by this paper
• TasvirEt: A benchmark dataset for automatic Turkish description generation from images

  2016cited by this paper
• Multi30K: Multilingual English-German Image Descriptions

  2016influential reference
• Unsupervised Visual Sense Disambiguation for Verbs using Multimodal Embeddings

  2016cited by this paper
• Re-evaluating Automatic Metrics for Image Captioning

  2016cited by this paper
• MMFeat: A Toolkit for Extracting Multi-Modal Features

  2016cited by this paper
• Multimodal Pivots for Image Caption Translation

  2016influential reference
• Edinburgh Neural Machine Translation Systems for WMT 16

  2016cited by this paper
• Multimodal Attention for Neural Machine Translation

  2016cited by this paper
• A Shared Task on Multimodal Machine Translation and Crosslingual Image Description

  2016influential reference
• Effective Approaches to Attention-based Neural Machine Translation

  2015cited by this paper
• Can machine translation systems be evaluated by the crowd alone

  2015cited by this paper
• Multi-Language Image Description with Neural Sequence Models

  2015cited by this paper
• Neural Machine Translation of Rare Words with Subword Units

  2015cited by this paper
• Bridge Correlational Neural Networks for Multilingual Multimodal Representation Learning

  2015cited by this paper
• Microsoft COCO Captions: Data Collection and Evaluation Server

  2015influential reference
• Deep Residual Learning for Image Recognition

  2015influential reference
• Multilingual Image Description with Neural Sequence Models

  2015cited by this paper
• Rethinking the Inception Architecture for Computer Vision

  2015cited by this paper
• Neural Machine Translation by Jointly Learning to Align and Translate

  2014cited by this paper
• ImageNet Large Scale Visual Recognition Challenge

  2014cited by this paper
• From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions

  2014cited by this paper
• Meteor Universal: Language Specific Translation Evaluation for Any Target Language

  2014influential reference
• TUHOI: Trento Universal Human Object Interaction Dataset

  2014cited by this paper
• Comparing Automatic Evaluation Measures for Image Description

  2014cited by this paper
• Microsoft COCO: Common Objects in Context

  2014influential reference
• SdeWaC - A Corpus of Parsable Sentences from the Web

  2013cited by this paper
• Parallel Data, Tools and Interfaces in OPUS

  2012cited by this paper
• Japanese and Korean voice search

  2012cited by this paper
• Better Hypothesis Testing for Statistical Machine Translation: Controlling for Optimizer Instability

  2011cited by this paper
• ImageNet: A large-scale hierarchical image database

  2009cited by this paper
• Moses: Open Source Toolkit for Statistical Machine Translation

  2007influential reference
• A Study of Translation Edit Rate with Targeted Human Annotation

  2006influential reference
• A STUDY OF TRANSLATION ERROR RATE WITH TARGETED HUMAN ANNOTATION

  2005influential reference
• Minimum Error Rate Training in Statistical Machine Translation

  2003cited by this paper
• Bleu: a Method for Automatic Evaluation of Machine Translation

  2002influential reference

• Multilingual-To-Multimodal (M2M): Unlocking New Languages with Monolingual Text

  2026cites this paper
• Is Multimodality Still Required for Multimodal Machine Translation? A Case Study on English and Italian

  2025cites this paper
• Multimodal Machine Translation with Text-Image In-depth Questioning

  2025influential citation
• TopicVD: A Topic-Based Dataset of Video-Guided Multimodal Machine Translation for Documentaries

  2025cites this paper
• MultiScript30k: Leveraging Multilingual Embeddings to Extend Cross Script Parallel Data

  2025cites this paper
• Multilingual Vision-Language Models, A Survey

  2025cites this paper
• Dual-branch Prompting for Multimodal Machine Translation

  2025cites this paper
• Multimodal Neural Machine Translation Based on Knowledge Distillation and Anti-Noise Interaction

  2025cites this paper
• Scaling Laws for Conditional Emergence of Multilingual Image Captioning via Generalization from Translation

  2025cites this paper
• GIIFT: Graph-guided Inductive Image-free Multimodal Machine Translation

  2025cites this paper
• ConECT Dataset: Overcoming Data Scarcity in Context-Aware E-Commerce MT

  2025cites this paper
• OmniFusion: Simultaneous Multilingual Multimodal Translations via Modular Fusion

  2025cites this paper
• LGVLM-mIoT: A Lightweight Generative Visual-Language Model for Multilingual IoT Applications

  2025cites this paper
• Can visual language models resolve textual ambiguity with visual cues? Let visual puns tell you!

  2024cites this paper
• Exploring the Necessity of Visual Modality in Multimodal Machine Translation using Authentic Datasets

  2024cites this paper
• Dose multimodal machine translation can improve translation performance?

  2024cites this paper
• Mitigating Translationese in Low-resource Languages: The Storyboard Approach

  2024cites this paper
• ELDA: Enhancing Multi-Modal Machine Translation via Large Language Model-Driven Data Augmentation

  2024cites this paper
• Language-based machine perception: linguistic perspectives on the compilation of captioning datasets

  2024cites this paper
• Soul-Mix: Enhancing Multimodal Machine Translation with Manifold Mixup

  2024influential citation
• Towards Zero-Shot Multimodal Machine Translation

  2024influential citation
• DejaVu: Disambiguation evaluation dataset for English-JApanese machine translation on VisUal information

  2024cites this paper
• Enhancing Translation Quality by Leveraging Semantic Diversity in Multimodal Machine Translation

  2024cites this paper
• Image captioning in different languages

  2024cites this paper
• Semantic similarity on multimodal data: A comprehensive survey with applications

  2024cites this paper
• Relay Decoding: Concatenating Large Language Models for Machine Translation

  2024cites this paper
• Framed Multi30K: A Frame-Based Multimodal-Multilingual Dataset

  2024influential citation
• A Survey on Multi-modal Machine Translation: Tasks, Methods and Challenges

  2024cites this paper
• Encoder–Decoder Calibration for Multimodal Machine Translation

  2024cites this paper
• mOSCAR: A Large-scale Multilingual and Multimodal Document-level Corpus

  2024cites this paper
• Benchmarking Visually-Situated Translation of Text in Natural Images

  2024cites this paper
• VisTFC: Vision-guided target-side future context learning for neural machine translation

  2024cites this paper
• Generalization Algorithm of Multimodal Pre-Training Model Based on Graph-Text Self-Supervised Training

  2023cites this paper
• Cross-modal text and visual generation: A systematic review. Part 1: Image to text

  2023cites this paper
• Enabling Unsupervised Neural Machine Translation with Word-level Visual Representations

  2023cites this paper
• Video-Helpful Multimodal Machine Translation

  2023cites this paper
• Parameter-Efficient Cross-lingual Transfer of Vision and Language Models via Translation-based Alignment

  2023cites this paper
• Visual Prediction Improves Zero-Shot Cross-Modal Machine Translation

  2023cites this paper
• BIG-C: a Multimodal Multi-Purpose Dataset for Bemba

  2023cites this paper
• Adapting Grounded Visual Question Answering Models to Low Resource Languages

  2023cites this paper
• Visual Question Generation in Bengali

  2023cites this paper
• Augmenters at SemEval-2023 Task 1: Enhancing CLIP in Handling Compositionality and Ambiguity for Zero-Shot Visual WSD through Prompt Augmentation and Text-To-Image Diffusion

  2023cites this paper
• Multimodal Translation Methodology for Intangible Cultural Heritage: A Case Study of Pingxiang Nuo Culture

  2023cites this paper
• Choosing What to Mask: More Informed Masking for Multimodal Machine Translation

  2023cites this paper
• Bridging the Gap between Synthetic and Authentic Images for Multimodal Machine Translation

  2023cites this paper
• Multimodal Neural Machine Translation Using Synthetic Images Transformed by Latent Diffusion Model

  2023cites this paper
• Layer-Level Progressive Transformer With Modality Difference Awareness for Multi-Modal Neural Machine Translation

  2023cites this paper
• Translation-Enhanced Multilingual Text-to-Image Generation

  2023cites this paper
• A Survey of Vision-Language Pre-training from the Lens of Multimodal Machine Translation

  2023cites this paper
• Babel-ImageNet: Massively Multilingual Evaluation of Vision-and-Language Representations

  2023cites this paper
• On the Copying Problem of Unsupervised NMT: A Training Schedule with a Language Discriminator Loss

  2023cites this paper
• DAS-CL: Towards Multimodal Machine Translation via Dual-Level Asymmetric Contrastive Learning

  2023cites this paper
• Analyzing Challenges in Neural Machine Translation for Software Localization

  2023influential citation
• Language Model Alignment with Elastic Reset

  2023influential citation
• PR-MCS: Perturbation Robust Metric for MultiLingual Image Captioning

  2023influential citation
• Unifying Cross-Lingual and Cross-Modal Modeling Towards Weakly Supervised Multilingual Vision-Language Pre-training

  2023cites this paper
• Visual language integration: A survey and open challenges

  2023cites this paper
• Contrastive Adversarial Training for Multi-Modal Machine Translation

  2023cites this paper
• Extension Multi30K: Multimodal Dataset for Integrated Vision and Language Research in Ukrainian

  2023cites this paper
• Incorporating Probing Signals into Multimodal Machine Translation via Visual Question-Answering Pairs

  2023cites this paper
• Tackling Ambiguity with Images: Improved Multimodal Machine Translation and Contrastive Evaluation

  2022cites this paper
• VALHALLA: Visual Hallucination for Machine Translation

  2022cites this paper
• A Subspace-Based Analysis of Structured and Unstructured Representations in Image-Text Retrieval

  2022cites this paper
• Representing Context in FrameNet: A Multidimensional, Multimodal Approach

  2022cites this paper
• On Vision Features in Multimodal Machine Translation

  2022cites this paper
• Multilingual Multimodality: A Taxonomical Survey of Datasets, Techniques, Challenges and Opportunities

  2022cites this paper
• Delving Deeper into Cross-lingual Visual Question Answering

  2022cites this paper
• ERNIE-UniX2: A Unified Cross-lingual Cross-modal Framework for Understanding and Generation

  2022cites this paper
• EMMT: A simultaneous eye-tracking, 4-electrode EEG and audio corpus for multi-modal reading and translation scenarios

  2022cites this paper
• LVP-M3: Language-aware Visual Prompt for Multilingual Multimodal Machine Translation

  2022cites this paper
• GLAMI-1M: A Multilingual Image-Text Fashion Dataset

  2022cites this paper
• Noise-robust Cross-modal Interactive Learning with Text2Image Mask for Multi-modal Neural Machine Translation

  2022cites this paper
• VISA: An Ambiguous Subtitles Dataset for Visual Scene-aware Machine Translation

  2022cites this paper
• Efficient Multilingual Multi-modal Pre-training through Triple Contrastive Loss

  2022cites this paper
• Supervised Visual Attention for Simultaneous Multimodal Machine Translation

  2022influential citation
• Distill The Image to Nowhere: Inversion Knowledge Distillation for Multimodal Machine Translation

  2022cites this paper
• Sentence Ambiguity, Grammaticality and Complexity Probes

  2022cites this paper
• Neural Natural Language Generation: A Survey on Multilinguality, Multimodality, Controllability and Learning

  2022cites this paper
• Contrastive Learning Based Visual Representation Enhancement for Multimodal Machine Translation

  2022cites this paper
• IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages

  2022cites this paper
• Bloom Library: Multimodal Datasets in 300+ Languages for a Variety of Downstream Tasks

  2022influential citation
• Multimodal Neural Machine Translation with Search Engine Based Image Retrieval

  2022cites this paper
• Analysing the Correlation between Lexical Ambiguity and Translation Quality in a Multimodal Setting using WordNet

  2022cites this paper
• Towards Multi-Lingual Visual Question Answering

  2022cites this paper
• HybridVocab: Towards Multi-Modal Machine Translation via Multi-Aspect Alignment

  2022cites this paper
• Multilingual and Multimodal Learning for Brazilian Portuguese

  2022cites this paper
• Pre-Trained Word Embedding and Language Model Improve Multimodal Machine Translation: A Case Study in Multi30K

  2022cites this paper
• Bootstrapping a high quality multilingual multimodal dataset for Bletchley

  2022cites this paper
• The Case for Perspective in Multimodal Datasets

  2022cites this paper
• Flow-Adapter Architecture for Unsupervised Machine Translation

  2022cites this paper
• Utilizing Language-Image Pretraining for Efficient and Robust Bilingual Word Alignment

  2022cites this paper
• A Study of Re-generating Sentences Given Similar Sentences that Cover Them on the Level of Form and Meaning

  2022cites this paper
• Multimodal Neural Machine Translation for Mongolian to Chinese

  2022cites this paper
• Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Dataset

  2022influential citation
• Training Heterogeneous Features in Sequence to Sequence Tasks: Latent Enhanced Multi-filter Seq2Seq Model

  2021cites this paper
• Assorted Attention Network for Cross-Lingual Language-to-Vision Retrieval

  2021cites this paper
• Guiding Visual Question Generation

  2021cites this paper
• Retrieve Fast, Rerank Smart: Cooperative and Joint Approaches for Improved Cross-Modal Retrieval

  2021cites this paper
• Cultural and Geographical Influences on Image Translatability of Words across Languages

  2021cites this paper
• LightningDOT: Pre-training Visual-Semantic Embeddings for Real-Time Image-Text Retrieval

  2021influential citation