Multi30K: Multilingual English-German Image Descriptions

We introduce the Multi30K dataset to stimulate multilingual multimodal research. Recent advances in image description have been demonstrated on English-language datasets almost exclusively, but image description should not be limited to English. This dataset extends the Flickr30K dataset with i) German translations created by professional translators over a subset of the English descriptions, and ii) descriptions crowdsourced independently of the original English descriptions. We outline how the data can be used for multilingual image description and multimodal machine translation, but we anticipate the data will be useful for a broader range of tasks.

• Publication year

  2016

• Venue

  VL@ACL

• Publication date

  2016-05-02

• Fields of study

  Linguistics, Computer Science

• Identifiers
  DOI 10.18653/v1/W16-3210arXiv 1605.00459
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• A Shared Task on Multimodal Machine Translation and Crosslingual Image Description

  2016cited by this paper
• Multimodal Pivots for Image Caption Translation

  2016cited by this paper
• Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measures

  2016cited by this paper
• Multi-Language Image Description with Neural Sequence Models

  2015cited by this paper
• Image-Mediated Learning for Zero-Shot Cross-Lingual Document Retrieval

  2015cited by this paper
• Microsoft COCO Captions: Data Collection and Evaluation Server

  2015cited by this paper
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention

  2015cited by this paper
• Multilingual Image Description with Neural Sequence Models

  2015cited by this paper
• Sequence to Sequence Learning with Neural Networks

  2014cited by this paper
• Show and tell: A neural image caption generator

  2014cited by this paper
• From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions

  2014influential reference
• Neural Machine Translation by Jointly Learning to Align and Translate

  2014cited by this paper
• Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics

  2013cited by this paper
• Image Description using Visual Dependency Representations

  2013cited by this paper
• Images as Context in Statistical Machine Translation

  2012cited by this paper
• Collecting Image Annotations Using Amazon’s Mechanical Turk

  2010cited by this paper
• The IAPR TC-12 Benchmark: A New Evaluation Resource for Visual Information Systems

  2006cited by this paper
• A Hierarchical Phrase-Based Model for Statistical Machine Translation

  2005cited by this paper
• Statistical Phrase-Based Translation

  2003cited by this paper

• UML: uncertainty-aware and mutual learning for noise-robust cross-lingual cross-modal retrieval

  2026cites this paper
• Cross-Modal Diffusion on Pretrained Alignment Codebook for Multimodal Machine Translation

  2026cites this paper
• Using Deep Learning to Generate Semantically Correct Hindi Captions

  2026influential citation
• Recursive mutual-supervised multi-modal mask-variational fusion with domain-anchor alignment for domain-specific Multimodal Neural Machine Translation

  2026cites this paper
• Scalable Multilingual Multimodal Machine Translation with Speech-Text Fusion

  2026influential citation
• Multilingual-To-Multimodal (M2M): Unlocking New Languages with Monolingual Text

  2026cites this paper
• PolyModNet: Advanced positional encodings and ethical bias mitigation in adaptive multimodal fusion for multilingual language understanding

  2025cites this paper
• Multimodal Neural Machine Translation: A Survey of the State of the Art

  2025influential citation
• Dual-branch Prompting for Multimodal Machine Translation

  2025cites this paper
• Chitranuvad: Adapting Multi-lingual LLMs for Multimodal Translation

  2025cites this paper
• The Hidden Bloat in Machine Learning Systems

  2025cites this paper
• Imagination and Contemplation: A Balanced Framework for Semantic-Augmented Multimodal Machine Translation

  2025influential citation
• Resource Heterogeneity-Aware and Utilization-Enhanced Scheduling for Deep Learning Clusters

  2025cites this paper
• Hybrid Quantum-Classical Recurrent Neural Networks

  2025cites this paper
• Two-stage incremental semantic aggregation for domain-specific multimodal neural machine translation

  2025cites this paper
• A Multimodal Recaptioning Framework to Account for Perceptual Diversity Across Languages in Vision-Language Modeling

  2025cites this paper
• Transformer-Encoder Trees for Efficient Multilingual Machine Translation and Speech Translation

  2025cites this paper
• Make Imagination Clearer! Stable Diffusion-based Visual Imagination for Multimodal Machine Translation

  2025influential citation
• TowerVision: Understanding and Improving Multilinguality in Vision-Language Models

  2025influential citation
• A Picture is Worth a Thousand (Correct) Captions: A Vision-Guided Judge-Corrector System for Multimodal Machine Translation

  2025cites this paper
• SAFE-AKT: Kazakh Image-Text Retrieval via Semantic-Agnostic Feature Enhancement and Adaptive Knowledge Transfer

  2025cites this paper
• Large Multimodal Models for Low-Resource Languages: A Survey

  2025cites this paper
• Multilingual Vision-Language Models, A Survey

  2025cites this paper
• MultiScript30k: Leveraging Multilingual Embeddings to Extend Cross Script Parallel Data

  2025cites this paper
• LGVLM-mIoT: A Lightweight Generative Visual-Language Model for Multilingual IoT Applications

  2025cites this paper
• From Benchmark to Better Embeddings: Leveraging Synonym Substitution to Enhance Multimodal Models in Ukrainian

  2025cites this paper
• Beyond Decoder-only: Large Language Models Can be Good Encoders for Machine Translation

  2025cites this paper
• Dynamic Pricing for On-Demand DNN Inference in the Edge-AI Market

  2025cites this paper
• Non-Autoregressive Multimodal Machine Translation

  2025cites this paper
• Batch Speculative Decoding Done Right

  2025cites this paper
• Exploring In-Image Machine Translation with Real-World Background

  2025cites this paper
• SchoenbAt: Rethinking Attention with Polynomial basis

  2025cites this paper
• Multimodal Machine Translation with Visual Scene Graph Pruning

  2025cites this paper
• From Text to Multi-Modal: Advancing Low-Resource-Language Translation through Synthetic Data Generation and Cross-Modal Alignments

  2025cites this paper
• ConECT Dataset: Overcoming Data Scarcity in Context-Aware E-Commerce MT

  2025cites this paper
• GIIFT: Graph-guided Inductive Image-free Multimodal Machine Translation

  2025cites this paper
• M3-RAG: Unified Multimodal and Multilingual Retrieval-Augmented Generation

  2025cites this paper
• Interpreting Deep Neural Networks via Relative Activation-Deactivation Abstractions

  2025cites this paper
• AFRICAPTION: Establishing a New Paradigm for Image Captioning in African Languages

  2025cites this paper
• Distilling Multilingual Vision-Language Models: When Smaller Models Stay Multilingual

  2025influential citation
• SHIFT: Selected Helpful Informative Frame for Video-guided Machine Translation

  2025cites this paper
• VQA-Augmented Machine Translation with Cross-Modal Contrastive Learning

  2025cites this paper
• OmniFusion: Simultaneous Multilingual Multimodal Translations via Modular Fusion

  2025cites this paper
• Alternative positional encoding functions for neural transformers

  2025cites this paper
• Pay Attention Later: From Vector Space Diffusion to Linearithmic Spectral Phase-Locking

  2025cites this paper
• Dual-Grained Visual–Text Alignment for Multimodal Neural Machine Translation

  2025cites this paper
• Detect, Disambiguate, and Translate: On-Demand Visual Reasoning for Multimodal Machine Translation with Large Vision-Language Models

  2025cites this paper
• M2-VLP: Enhancing Multilingual Vision-Language Pre-Training via Multi-Grained Alignment

  2025cites this paper
• Decentralized Nonconvex Optimization under Heavy-Tailed Noise: Normalization and Optimal Convergence

  2025cites this paper
• Is Multimodality Still Required for Multimodal Machine Translation? A Case Study on English and Italian

  2025cites this paper
• Enhancing multimodal translation: Achieving consistency among visual information, source language and target language

  2025cites this paper
• TriFine: A Large-Scale Dataset of Vision-Audio-Subtitle for Tri-Modal Machine Translation and Benchmark with Fine-Grained Annotated Tags

  2025cites this paper
• Adaptive Width Neural Networks

  2025cites this paper
• Brain network science modelling of sparse neural networks enables Transformers and LLMs to perform as fully connected

  2025cites this paper
• Multimodal Neural Machine Translation Based on Knowledge Distillation and Anti-Noise Interaction

  2025cites this paper
• Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generation

  2025cites this paper
• A Collaborative Approach to Multimodal Machine Translation: VLM and LLM

  2025cites this paper
• A Benchmark for Multi-Lingual Vision-Language Learning in Remote Sensing Image Captioning

  2025cites this paper
• UICD: A new dataset and approach for urdu image captioning

  2025cites this paper
• IndicVisionBench: Benchmarking Cultural and Multilingual Understanding in VLMs

  2025cites this paper
• Scaling Laws for Conditional Emergence of Multilingual Image Captioning via Generalization from Translation

  2025influential citation
• Seeing Through Ambiguity: Effective Video-guided Machine Translation via Chaotic Fusion and Causally Aligned Spatio-temporal Attention

  2025cites this paper
• Multilingual Text-to-Image Person Retrieval via Bidirectional Relation Reasoning and Aligning

  2025cites this paper
• A Multimodal Recaptioning Framework to Account for Perceptual Diversity in Multilingual Vision-Language Modeling

  2025cites this paper
• Unravelling the semantic mysteries of transformers layer by layer

  2025cites this paper
• Memory Reviving, Continuing Learning and Beyond: Evaluation of Pre-trained Encoders and Decoders for Multimodal Machine Translation

  2025influential citation
• TopicVD: A Topic-Based Dataset of Video-Guided Multimodal Machine Translation for Documentaries

  2025cites this paper
• Behind Maya: Building a Multilingual Vision Language Model

  2025cites this paper
• Cross-Lingual Representation Alignment Through Contrastive Image-Caption Tuning

  2025cites this paper
• Parallel Layer Normalization for Universal Approximation

  2025cites this paper
• Chameleon: A Multimodal Learning Framework Robust to Missing Modalities

  2025cites this paper
• AraTraditions10k bridging cultures with a comprehensive dataset for enhanced cross lingual image annotation retrieval and tagging

  2025cites this paper
• Perturbative Gradient Training: A novel training paradigm for bridging the gap between deep neural networks and physical reservoir computing

  2025cites this paper
• Plug-in and Fine-tuning: Bridging the Gap between Small Language Models and Large Language Models

  2025cites this paper
• Multimodal Machine Translation with Text-Image In-depth Questioning

  2025influential citation
• AuroraLong: Bringing RNNs Back to Efficient Open-Ended Video Understanding

  2025cites this paper
• VisTFC: Vision-guided target-side future context learning for neural machine translation

  2024cites this paper
• Embracing Language Inclusivity and Diversity in CLIP through Continual Language Learning

  2024cites this paper
• Encoder–Decoder Calibration for Multimodal Machine Translation

  2024cites this paper
• Deep Learning Approaches for Image Captioning: Opportunities, Challenges and Future Potential

  2024cites this paper
• A 77.79 GOPs/W Retentive Network FPGA Inference Accelerator with Optimized Workload

  2024cites this paper
• A Multimodal In-Context Tuning Approach for E-Commerce Product Description Generation

  2024cites this paper
• SITH: Semantic Interpreter for Transformer Hierarchy

  2024cites this paper
• Textual Out-of-Distribution Data Detection Based on Granular Dictionary

  2024cites this paper
• A Categorical Particle Swarm Optimization for Hyperparameter Optimization in Low-Resource Transformer-based Machine Translation

  2024cites this paper
• Homeostasis and Sparsity in Transformer

  2024cites this paper
• Multi-Head Encoding for Extreme Label Classification

  2024cites this paper
• Assessing the Role of Imagery in Multimodal Machine Translation

  2024cites this paper
• No Culture Left Behind: ArtELingo-28, a Benchmark of WikiArt with Captions in 28 Languages

  2024cites this paper
• Context-Informed Machine Translation of Manga using Multimodal Large Language Models

  2024cites this paper
• Curriculum Consistency Learning for Conditional Sentence Generation

  2024cites this paper
• ELDA: Enhancing Multi-Modal Machine Translation via Large Language Model-Driven Data Augmentation

  2024cites this paper
• Can visual language models resolve textual ambiguity with visual cues? Let visual puns tell you!

  2024influential citation
• Multimodal LLM Enhanced Cross-lingual Cross-modal Retrieval

  2024cites this paper
• Quantifying the Gaps Between Translation and Native Perception in Training for Multimodal, Multilingual Retrieval

  2024influential citation
• Brotherhood at WMT 2024: Leveraging LLM-Generated Contextual Conversations for Cross-Lingual Image Captioning

  2024cites this paper
• Visual Pivoting Unsupervised Multimodal Machine Translation in Low-Resource Distant Language Pairs

  2024cites this paper
• Benchmarking Visually-Situated Translation of Text in Natural Images

  2024cites this paper
• Benchmarking Multimodal Models for Ukrainian Language Understanding Across Academic and Cultural Domains

  2024cites this paper
• Straggler-Aware Gradient Aggregation for Large-Scale Distributed Deep Learning System

  2024cites this paper