Fluency-Guided Cross-Lingual Image Captioning

Image captioning has so far been explored mostly in English, as most available datasets are in this language. However, the application of image captioning should not be restricted by language. Only few studies have been conducted for image captioning in a cross-lingual setting. Different from these works that manually build a dataset for a target language, we aim to learn a cross-lingual captioning model fully from machine-translated sentences. To conquer the lack of fluency in the translated sentences, we propose in this paper a fluency-guided learning framework. The framework comprises a module to automatically estimate the fluency of the sentences and another module to utilize the estimated fluency scores to effectively train an image captioning model for the target language. As experiments on two bilingual (English-Chinese) datasets show, our approach improves both fluency and relevance of the generated captions in Chinese, but without using any manually written sentences from the target language.

• Publication year

  2017

• Venue

  ACM Multimedia

• Publication date

  2017-08-15

• Fields of study

  Linguistics, Computer Science

• Identifiers
  DOI 10.1145/3123266.3123366arXiv 1708.04390
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Image Captioning with Deep Bidirectional LSTMs

  2016cited by this paper
• Adding Chinese Captions to Images

  2016influential reference
• Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measures

  2016cited by this paper
• Boosting Image Captioning with Attributes

  2016cited by this paper
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding

  2016cited by this paper
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation

  2016cited by this paper
• Deep Recurrent Models with Fast-Forward Connections for Neural Machine Translation

  2016cited by this paper
• Generating Affective Captions using Concept And Syntax Transition Networks

  2016cited by this paper
• Image Captioning with both Object and Scene Information

  2016cited by this paper
• Cross-Lingual Image Caption Generation

  2016cited by this paper
• Self-Critical Sequence Training for Image Captioning

  2016cited by this paper
• YFCC100M

  2015cited by this paper
• Deep Residual Learning for Image Recognition

  2015cited by this paper
• Multilingual Image Description with Neural Sequence Models

  2015cited by this paper
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention

  2015cited by this paper
• BosonNLP: An Ensemble Approach for Word Segmentation and POS Tagging

  2015cited by this paper
• What Value Do Explicit High Level Concepts Have in Vision to Language Problems?

  2015cited by this paper
• Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question

  2015cited by this paper
• Deep visual-semantic alignments for generating image descriptions

  2014cited by this paper
• CIDEr: Consensus-based image description evaluation

  2014cited by this paper
• From captions to visual concepts and back

  2014cited by this paper
• Neural Machine Translation by Jointly Learning to Align and Translate

  2014cited by this paper
• A Fast and Accurate Dependency Parser using Neural Networks

  2014cited by this paper
• Microsoft COCO: Common Objects in Context

  2014cited by this paper
• Show and tell: A neural image caption generator

  2014cited by this paper
• Grounded Compositional Semantics for Finding and Describing Images with Sentences

  2014cited by this paper
• From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions

  2014cited by this paper
• Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)

  2014cited by this paper
• Meteor Universal: Language Specific Translation Evaluation for Any Target Language

  2014cited by this paper
• Adam: A Method for Stochastic Optimization

  2014influential reference
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models

  2014cited by this paper
• DeViSE: A Deep Visual-Semantic Embedding Model

  2013cited by this paper
• Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics

  2013influential reference
• Statistical Machine Translation with Readability Constraints

  2013influential reference
• Learning cross-modality similarity for multinomial data

  2011cited by this paper
• Every Picture Tells a Story: Generating Sentences from Images

  2010cited by this paper
• Improving Translation Fluency with Search-Based Decoding and a Monolingual Statistical Machine Translation Model for Automatic Post-Editing

  2009cited by this paper
• Cost-Sensitive Learning vs. Sampling: Which is Best for Handling Unbalanced Classes with Unequal Error Costs?

  2007cited by this paper
• ROUGE: A Package for Automatic Evaluation of Summaries

  2004cited by this paper
• Bleu: a Method for Automatic Evaluation of Machine Translation

  2002influential reference
• Principles of Context-Based Machine Translation Evaluation

  2002cited by this paper
• The Foundations of Cost-Sensitive Learning

  2001cited by this paper
• Long Short-Term Memory

  1997cited by this paper

• DanQing: An Up-to-Date Large-Scale Chinese Vision-Language Pre-training Dataset

  2026cites this paper
• Reasoning text-to-image retrieval with large language models and digital twin representations

  2026cites this paper
• ObjEmbed: Towards Universal Multimodal Object Embeddings

  2026cites this paper
• Arbitrary Ratio Feature Compression via Next Token Prediction

  2026cites this paper
• Using Deep Learning to Generate Semantically Correct Hindi Captions

  2026influential citation
• Ta-CLIP: A Framework for Chinese Cross-Modal Retrieval Using Text-Aware and Contrastive Learning

  2025cites this paper
• AraTraditions10k bridging cultures with a comprehensive dataset for enhanced cross lingual image annotation retrieval and tagging

  2025cites this paper
• UICD: A new dataset and approach for urdu image captioning

  2025cites this paper
• Towards Cross-modal Retrieval in Chinese Cultural Heritage Documents: Dataset and Solution

  2025cites this paper
• Bridging Vision and Language: A CNN-Transformer Model for Image Captioning

  2025cites this paper
• A multilevel fine-grained feature fusion method for cross-modal image–text retrieval

  2025cites this paper
• HAN: Korean Heritage Augmented Narrative Visual-Language Description Dataset

  2025cites this paper
• GRAPE: Let GPRO Supervise Query Rewriting by Ranking for Retrieval

  2025cites this paper
• Pre-Trained CNN Architecture for Transformer-Based Image Caption Generation Model

  2025cites this paper
• Chinese Image Captioning Based on Deep Fusion Feature and Multi-Layer Feature Filtering Block

  2025cites this paper
• Large Vison-Language Foundation Model in Baidu AIGC Image Advertising

  2025cites this paper
• A Benchmark for Multi-Lingual Vision-Language Learning in Remote Sensing Image Captioning

  2025cites this paper
• Image captioning in Bengali language using visual attention

  2025cites this paper
• LGVLM-mIoT: A Lightweight Generative Visual-Language Model for Multilingual IoT Applications

  2025cites this paper
• Cross-modal Prompt-Driven Network for low-resource vision-to-language generation

  2025cites this paper
• Multilingual Diversity Improves Vision-Language Representations

  2024cites this paper
• CVLUE: A New Benchmark Dataset for Chinese Vision-Language Understanding Evaluation

  2024cites this paper
• LLM2CLIP: Powerful Language Model Unlocks Richer Visual Representation

  2024cites this paper
• Scaling Vison-Language Foundation Model to 12 Billion Parameters in Baidu Dynamic Image Advertising

  2024cites this paper
• Deep Learning Approaches for Image Captioning: Opportunities, Challenges and Future Potential

  2024cites this paper
• Video ReCap: Recursive Captioning of Hour-Long Videos

  2024cites this paper
• Framed Multi30K: A Frame-Based Multimodal-Multilingual Dataset

  2024cites this paper
• M2-Encoder: Advancing Bilingual Image-Text Understanding by Large-scale Efficient Pretraining

  2024influential citation
• Cross-Lingual Cross-Modal Retrieval With Noise-Robust Fine-Tuning

  2024cites this paper
• Image captioning in different languages

  2024cites this paper
• LLM2CLIP: Powerful Language Model Unlocks Richer Cross-Modality Representation

  2024cites this paper
• Embedded Heterogeneous Attention Transformer for Cross-Lingual Image Captioning

  2023cites this paper
• A comprehensive survey on image captioning: from handcrafted to deep learning-based techniques, a taxonomy and open research issues

  2023cites this paper
• Cross2StrA: Unpaired Cross-lingual Image Captioning with Cross-lingual Cross-modal Structure-pivoted Alignment

  2023influential citation
• Translation-Enhanced Multilingual Text-to-Image Generation

  2023cites this paper
• Enhancing Dynamic Image Advertising with Vision-Language Pre-training

  2023cites this paper
• Rapid Diffusion: Building Domain-Specific Text-to-Image Synthesizers with Fast Inference Speed

  2023cites this paper
• Pre-Trained CNN Architecture Analysis for Transformer-Based Indonesian Image Caption Generation Model

  2023cites this paper
• CITE: Compact Interactive TransformEr for Multilingual Image Captioning

  2023cites this paper
• A comprehensive survey on deep-learning-based visual captioning

  2023cites this paper
• Visual-linguistic-stylistic Triple Reward for Cross-lingual Image Captioning

  2023influential citation
• Intern VL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks

  2023cites this paper
• A Visual Attention-Based Model for Bengali Image Captioning

  2023cites this paper
• AGREE: Aligning Cross-Modal Entities for Image-Text Retrieval Upon Vision-Language Pre-trained Models

  2023influential citation
• ZeroNLG: Aligning and Autoencoding Domains for Zero-Shot Multimodal and Multilingual Natural Language Generation

  2023cites this paper
• When Pairs Meet Triplets: Improving Low-Resource Captioning via Multi-Objective Optimization

  2022cites this paper
• EAES: Effective Augmented Embedding Spaces for Text-based Image Captioning

  2022cites this paper
• Neural Natural Language Generation: A Survey on Multilinguality, Multimodality, Controllability and Learning

  2022cites this paper
• Zero and R2D2: A Large-scale Chinese Cross-modal Benchmark and A Vision-Language Framework

  2022influential citation
• Wukong: A 100 Million Large-scale Chinese Cross-modal Pre-training Benchmark

  2022influential citation
• CLEAR: Improving Vision-Language Navigation with Cross-Lingual, Environment-Agnostic Representations

  2022cites this paper
• Cross-Lingual Cross-Modal Retrieval with Noise-Robust Learning

  2022cites this paper
• CapNet: An Encoder-Decoder based Neural Network Model for Automatic Bangla Image Caption Generation

  2022cites this paper
• CCMB: A Large-scale Chinese Cross-modal Benchmark

  2022cites this paper
• Multilingual Multimodality: A Taxonomical Survey of Datasets, Techniques, Challenges and Opportunities

  2022cites this paper
• Chinese CLIP: Contrastive Vision-Language Pretraining in Chinese

  2022influential citation
• AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilities

  2022cites this paper
• Automatic Indonesian Image Captioning using CNN and Transformer-Based Model Approach

  2022cites this paper
• ARTIST: A Transformer-based Chinese Text-to-Image Synthesizer Digesting Linguistic and World Knowledge

  2022cites this paper
• IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages

  2022cites this paper
• Wukong: 100 Million Large-scale Chinese Cross-modal Pre-training Dataset and A Foundation Framework

  2022cites this paper
• ABID: Attention-Based Bengali Image Description

  2022cites this paper
• CJE-TIG: Zero-shot cross-lingual text-to-image generation by Corpora-based Joint Encoding

  2021cites this paper
• Self-Distillation for Few-Shot Image Captioning

  2021cites this paper
• Exploring Video Captioning Techniques: A Comprehensive Survey on Deep Learning Methods

  2021cites this paper
• A Hybridized Deep Learning Method for Bengali Image Captioning

  2021cites this paper
• Deep Image Captioning Survey: A Resource Availability Perspective

  2021cites this paper
• From Show to Tell: A Survey on Image Captioning

  2021cites this paper
• mTVR: Multilingual Moment Retrieval in Videos

  2021cites this paper
• Chinese description of videos incorporating multimodal features and attention mechanism

  2021cites this paper
• Bornon: Bengali Image Captioning with Transformer-Based Deep Learning Approach

  2021cites this paper
• Improving German Image Captions Using Machine Translation and Transfer Learning

  2021cites this paper
• A Rapid Review of Image Captioning

  2021cites this paper
• Assorted Attention Network for Cross-Lingual Language-to-Vision Retrieval

  2021cites this paper
• A Scientometric Visualization Analysis of Image Captioning Research from 2010 to 2020

  2021cites this paper
• From Show to Tell: A Survey on Deep Learning-Based Image Captioning

  2021cites this paper
• Object-Centric Unsupervised Image Captioning

  2021cites this paper
• Aligning Source Visual and Target Language Domains for Unpaired Video Captioning

  2021cites this paper
• A Visually-Grounded Parallel Corpus with Phrase-to-Region Linking

  2020influential citation
• Characterization and classification of semantic image-text relations

  2020cites this paper
• iCap: Interactive Image Captioning with Predictive Text

  2020influential citation
• Denoising Large-Scale Image Captioning from Alt-text Data Using Content Selection Models

  2020cites this paper
• Bengali Image Captioning with Visual Attention

  2020cites this paper
• Towards annotation-free evaluation of cross-lingual image captioning

  2020cites this paper
• An Improved Method of Cross-Lingual Image Captioning Based on Fluency-Guided

  2020cites this paper
• Vision to Language: Methods, Metrics and Datasets

  2020cites this paper
• UNISON: Unpaired Cross-Lingual Image Captioning

  2020cites this paper
• Unsupervised Cross-lingual Image Captioning

  2020cites this paper
• Evaluation of Text Generation: A Survey

  2020cites this paper
• Cross-Lingual Image Caption Generation Based on Visual Attention Model

  2020cites this paper
• Human Object Interaction Detection via Multi-level Conditioned Network

  2020cites this paper
• Deep Image Captioning: An Overview

  2019cites this paper
• VaTeX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research

  2019cites this paper
• Know More Say Less: Image Captioning Based on Scene Graphs

  2019cites this paper
• Deep learning approach for Image captioning in Hindi language

  2019influential citation
• Deep Learning for Video Retrieval by Natural Language

  2019cites this paper
• MULE: Multimodal Universal Language Embedding

  2019cites this paper
• Oboyob: A sequential-semantic Bengali image captioning engine

  2019cites this paper
• Unpaired Cross-lingual Image Caption Generation with Self-Supervised Rewards

  2019influential citation
• Improving Captioning for Low-Resource Languages by Cycle Consistency

  2019cites this paper