Exploring Models and Data for Image Question Answering

This work aims to address the problem of image-based question-answering (QA) with new models and datasets. In our work, we propose to use neural networks and visual semantic embeddings, without intermediate stages such as object detection and image segmentation, to predict answers to simple questions about images. Our model performs 1.8 times better than the only published results on an existing image QA dataset. We also present a question generation algorithm that converts image descriptions, which are widely available, into QA form. We used this algorithm to produce an order-of-magnitude larger dataset, with more evenly distributed answers. A suite of baseline results on this new dataset are also presented.

• Publication year

  2015

• Venue

  Neural Information Processing Systems

• Publication date

  2015-05-08

• Fields of study

  Computer Science

• Identifiers
  arXiv 1505.02074
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Phrase-based Image Captioning

  2015cited by this paper
• VQA: Visual Question Answering

  2015cited by this paper
• Microsoft COCO Captions: Data Collection and Evaluation Server

  2015cited by this paper
• Learning to Answer Questions from Image Using Convolutional Neural Network

  2015cited by this paper
• Ask Your Neurons: A Neural-Based Approach to Answering Questions about Images

  2015cited by this paper
• Exploring Nearest Neighbor Approaches for Image Captioning

  2015cited by this paper
• Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question

  2015influential reference
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention

  2015cited by this paper
• Explain Images with Multimodal Recurrent Neural Networks

  2014cited by this paper
• Towards a Visual Turing Challenge

  2014influential reference
• Show and tell: A neural image caption generator

  2014influential reference
• Deep Fragment Embeddings for Bidirectional Image Sentence Mapping

  2014cited by this paper
• ImageNet Large Scale Visual Recognition Challenge

  2014cited by this paper
• A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Input

  2014influential reference
• Long-term recurrent convolutional networks for visual recognition and description

  2014cited by this paper
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models

  2014cited by this paper
• From captions to visual concepts and back

  2014cited by this paper
• Microsoft COCO: Common Objects in Context

  2014influential reference
• Very Deep Convolutional Networks for Large-Scale Image Recognition

  2014cited by this paper
• Fisher Vectors Derived from Hybrid Gaussian-Laplacian Mixture Models for Image Annotation

  2014cited by this paper
• Learning a Recurrent Visual Representation for Image Caption Generation

  2014cited by this paper
• Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics

  2013cited by this paper
• DeViSE: A Deep Visual-Semantic Embedding Model

  2013cited by this paper
• Efficient Estimation of Word Representations in Vector Space

  2013cited by this paper
• Indoor Segmentation and Support Inference from RGBD Images

  2012cited by this paper
• Im2Text: Describing Images Using 1 Million Captioned Photographs

  2011cited by this paper
• Learning Dependency-Based Compositional Semantics

  2011cited by this paper
• NLTK: The Natural Language Toolkit

  2006cited by this paper
• Accurate Unlexicalized Parsing

  2003cited by this paper
• Book Reviews: WordNet: An Electronic Lexical Database

  1999cited by this paper
• Long Short-Term Memory

  1997cited by this paper
• Verb Semantics and Lexical Selection

  1994cited by this paper
• Conditions on transformations

  1971cited by this paper

• ERFC: Energy-Aware Reinforcement Feedback Calibration for Zero-Shot Captioning

  2026cites this paper
• Improving Episodic Few-shot Visual Question Answering via Spatial and Frequency Domain Dual-calibration

  2026cites this paper
• Prompt-Aware Adapter: Learning Adaptive Visual Tokens for Multimodal Large Language Models

  2026influential citation
• VAMOS: A Hierarchical Vision-Language-Action Model for Capability-Modulated and Steerable Navigation

  2025cites this paper
• ReasonVQA: A Multi-hop Reasoning Benchmark with Structural Knowledge for Visual Question Answering

  2025cites this paper
• A Comprehensive Review of Multimodal Visual Representation Learning: Tracing the Evolution from CNNs to Transformers and Beyond

  2025cites this paper
• An Overview of Robot Embodied Intelligence Based on Multimodal Models: Tasks, Models, and System Schemes

  2025cites this paper
• The compositions, characteristics, health benefits and applications of anthocyanins in Brassica crops

  2025cites this paper
• Talking to the brain: Using Large Language Models as Proxies to Model Brain Semantic Representation

  2025cites this paper
• Unsupervised Domain Adaptive Visual Question Answering in the Era of Multi-Modal Large Language Models

  2025cites this paper
• Not All Clients Are Equal: Collaborative Model Personalization on Heterogeneous Multi-Modal Clients

  2025cites this paper
• A Multi-Modal Attentive Framework That Can Interpret Text (MMAT)

  2025cites this paper
• Interactive Environment-Aware Planning System and Dialogue for Social Robots in Early Childhood Education

  2025cites this paper
• Enhancing Scientific Visual Question Answering via Vision-Caption aware Supervised Fine-Tuning

  2025cites this paper
• A multimodal transformer-based visual question answering method integrating local and global information

  2025cites this paper
• One More Modality: Does Abstract Meaning Representation Benefit Visual Question Answering?

  2025cites this paper
• LiGT: layout-infused generative transformer for visual question answering on Vietnamese receipts

  2025cites this paper
• ASAM: Asynchronous self-attention model for visual question answering

  2025cites this paper
• End-to-End 3D point cloud question answering via state space models

  2025cites this paper
• Improving chart question answering through integration of figure captions and visual–textual co-attention

  2025cites this paper
• Not All Clients Are Equal: Personalized Federated Learning on Heterogeneous Multi-Modal Clients

  2025cites this paper
• RAG-Check: Evaluating Multimodal Retrieval Augmented Generation Performance

  2025cites this paper
• Text-VQA Aug: Pipelined Harnessing of Large Multimodal Models for Automated Synthesis

  2025cites this paper
• Efficient GPT-4V level multimodal large language model for deployment on edge devices

  2025cites this paper
• Inferential and Commonsense Visual Question Generation

  2025cites this paper
• Instruction-Grounded Visual Projectors for Continual Learning of Generative Vision-Language Models

  2025influential citation
• Health-oriented Multimodal Food Question Answering with Implicit and Explicit Knowledge

  2025cites this paper
• JRDB-Reasoning: A Difficulty-Graded Benchmark for Visual Reasoning in Robotics

  2025cites this paper
• Advancing Visual Question Answering Through Multimodal Fusion and Deep Learning Techniques

  2025cites this paper
• VQArt-Bench: A Semantically Rich VQA Benchmark for Art and Cultural Heritage

  2025cites this paper
• A visual question answering method based on task decomposition

  2025cites this paper
• Understanding question-answering systems: Evolution, applications, trends, and challenges

  2025cites this paper
• Make "V" and "Q" Inseparable: Deliberately Dual-Channel Adversarial Learning for Robust Visual Question Answering

  2025cites this paper
• MSB-VQA: Overcoming multiple source biases for robust visual question answering

  2025cites this paper
• MLLM-Selector: Necessity and Diversity-driven High-Value Data Selection for Enhanced Visual Instruction Tuning

  2025cites this paper
• Towards Improved Text-Aligned Codebook Learning: Multi-Hierarchical Codebook-Text Alignment with Long Text

  2025cites this paper
• MindLLM: A Subject-Agnostic and Versatile Model for fMRI-to-Text Decoding

  2025cites this paper
• VQA and Visual Reasoning: An overview of approaches, datasets, and future direction

  2025cites this paper
• Megrez-Omni Technical Report

  2025cites this paper
• ExDDV: A New Dataset for Explainable Deepfake Detection in Video

  2025cites this paper
• M2-omni: Advancing Omni-MLLM for Comprehensive Modality Support with Competitive Performance

  2025cites this paper
• OWLViz: An Open-World Benchmark for Visual Question Answering

  2025cites this paper
• Fine-tuning Multimodal Transformers on Edge: A Parallel Split Learning Approach

  2025cites this paper
• Consistency Conditioned Memory Augmented Dynamic Diagnosis Model for Medical Visual Question Answering

  2025influential citation
• Performance vs. Complexity Comparative Analysis of Multimodal Bilinear Pooling Fusion Approaches for Deep Learning-Based Visual Arabic-Question Answering Systems

  2025cites this paper
• FedVLMBench: Benchmarking Federated Fine-Tuning of Vision-Language Models

  2025cites this paper
• FineVision: Open Data Is All You Need

  2025cites this paper
• LocateAnything3D: Vision-Language 3D Detection with Chain-of-Sight

  2025cites this paper
• Why is a Bird's Caption a Good Demonstration? Towards Effective Multimodal In-Context Learning without Dedicated Data

  2025cites this paper
• CFMMC-Align: Coarse-Fine Multi-Modal Contrastive Alignment Network for Traffic Event Video Question Answering

  2024cites this paper
• An Automated Mechanism for Retrospective Event Discovery via Visual Query Responses

  2024cites this paper
• Multimodal Deep Learning for Low-Resource Settings: A Vector Embedding Alignment Approach for Healthcare Applications

  2024cites this paper
• Deep learning-based bridge damage cause estimation from multiple images using visual question answering

  2024cites this paper
• Vision transformer-based visual language understanding of the construction process

  2024cites this paper
• Subjective Scoring Framework for VQA Models in Autonomous Driving

  2024cites this paper
• PGCL: Prompt guidance and self-supervised contrastive learning-based method for Visual Question Answering

  2024cites this paper
• Mix-tower: Light visual question answering framework based on exclusive self-attention mechanism

  2024cites this paper
• What matters when building vision-language models?

  2024influential citation
• Prompt-Aware Adapter: Towards Learning Adaptive Visual Tokens for Multimodal Large Language Models

  2024influential citation
• Cross-Modal Feature Distribution Calibration for Few-Shot Visual Question Answering

  2024influential citation
• Elysium: Exploring Object-level Perception in Videos via MLLM

  2024cites this paper
• A Novel Pretrained General-purpose Vision Language Model for the Vietnamese Language

  2024cites this paper
• Grounding Language Models for Visual Entity Recognition

  2024cites this paper
• LLCP: Learning Latent Causal Processes for Reasoning-based Video Question Answer

  2024cites this paper
• Beyond Text: Frozen Large Language Models in Visual Signal Comprehension

  2024influential citation
• Modular dual-stream visual fusion network for visual question answering

  2024cites this paper
• Assistive Image Annotation Systems with Deep Learning and Natural Language Capabilities: A Review

  2024cites this paper
• LawViVQA: A Visual Question Answering Dataset for Vietnamese Legal Content

  2024cites this paper
• COCO is “ALL” You Need for Visual Instruction Fine-tuning

  2024cites this paper
• VL-Meta: Vision-Language Models for Multimodal Meta-Learning

  2024cites this paper
• Exploring Visual Multiple-Choice Question Answering with Pre-trained Vision-Language Models

  2024cites this paper
• Dynamic Transformer Architecture for Continual Learning of Multimodal Tasks

  2024cites this paper
• Deep learning–radiomics integrated noninvasive detection of epidermal growth factor receptor mutations in non-small cell lung cancer patients

  2024cites this paper
• Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey

  2024cites this paper
• Enhancing Cross-Modal Contextual Congruence for Crowdfunding Success using Knowledge-infused Learning

  2024cites this paper
• A Comprehensive Survey on Visual Question Answering Datasets and Algorithms

  2024cites this paper
• Learning to Ask Denotative and Connotative Questions for Knowledge-based VQA

  2024cites this paper
• MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scale

  2024cites this paper
• FTASD - A Fine Tuning Approach for Stable Diffusion Models

  2024cites this paper
• Show Me What and Where has Changed? Question Answering and Grounding for Remote Sensing Change Detection

  2024cites this paper
• Deep multimodal relational reasoning and guided attention for chart question answering

  2024cites this paper
• ChitroJera: A Regionally Relevant Visual Question Answering Dataset for Bangla

  2024cites this paper
• Towards Foundation Models for 3D Vision: How Close are We?

  2024cites this paper
• EfficientEQA: An Efficient Approach to Open-Vocabulary Embodied Question Answering

  2024cites this paper
• To Boost Zero-Shot Generalization for Embodied Reasoning With Vision-Language Pre-Training

  2024cites this paper
• VADS: Visuo-Adaptive DualStrike attack on visual question answer

  2024cites this paper
• GEXMERT: Geometrically enhanced cross-modality encoder representations from transformers inspired by higher-order visual percepts

  2024cites this paper
• SimpsonsVQA: Enhancing Inquiry-Based Learning with a Tailored Dataset

  2024cites this paper
• A Preliminary Look at Generative AI for the Creation of Abstract Verbal-to-visual Analogies

  2024cites this paper
• Building and better understanding vision-language models: insights and future directions

  2024cites this paper
• Revisiting Multi-Modal LLM Evaluation

  2024cites this paper
• Diverse Visual Question Generation Based on Multiple Objects Selection

  2024cites this paper
• Video question answering via traffic knowledge database and question classification

  2024cites this paper
• Overcoming language priors in visual question answering with cumulative learning strategy

  2024cites this paper
• Towards Flexible Evaluation for Generative Visual Question Answering

  2024influential citation
• EQA-MX: Embodied Question Answering using Multimodal Expression

  2024cites this paper
• Visual Question Answering

  2024cites this paper
• Deep Multimodal Data Fusion

  2024influential citation
• Advanced Visual and Textual Co-context Aware Attention Network with Dependent Multimodal Fusion Block for Visual Question Answering

  2024cites this paper
• Decoding Visual Question Answering Methodologies: Unveiling Applications in Multimodal Learning Frameworks

  2024cites this paper