Teaching Compositionality to CNNs

Convolutional neural networks (CNNs) have shown great success in computer vision, approaching human-level performance when trained for specific tasks via application-specific loss functions. In this paper, we propose a method for augmenting and training CNNs so that their learned features are compositional. It encourages networks to form representations that disentangle objects from their surroundings and from each other, thereby promoting better generalization. Our method is agnostic to the specific details of the underlying CNN to which it is applied and can in principle be used with any CNN. As we show in our experiments, the learned representations lead to feature activations that are more localized and improve performance over non-compositional baselines in object recognition tasks.

• Publication year

  2017

• Venue

  Computer Vision and Pattern Recognition

• Publication date

  2017-06-14

• Fields of study

  Computer Science

• Identifiers
  DOI 10.1109/CVPR.2017.85arXiv 1706.04313
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Deep learning for neural networks

  2018cited by this paper
• Towards deep compositional networks

  2016influential reference
• Learning with Side Information through Modality Hallucination

  2016cited by this paper
• The Curious Robot: Learning Visual Representations via Physical Interactions

  2016cited by this paper
• Attend, Infer, Repeat: Fast Scene Understanding with Generative Models

  2016cited by this paper
• Texture Segregation Causes Early Figure Enhancement and Later Ground Suppression in Areas V1 and V4 of Visual Cortex

  2016cited by this paper
• Object-Centric Representation Learning from Unlabeled Videos

  2016cited by this paper
• Learning to See by Moving

  2015cited by this paper
• Exploring Person Context and Local Scene Context for Object Detection

  2015cited by this paper
• Is object localization for free? - Weakly-supervised learning with convolutional neural networks

  2015influential reference
• Slow and Steady Feature Analysis: Higher Order Temporal Coherence in Video

  2015cited by this paper
• Hypercolumns for object segmentation and fine-grained localization

  2014cited by this paper
• Show and tell: A neural image caption generator

  2014cited by this paper
• Going deeper with convolutions

  2014cited by this paper
• Semantic part segmentation using compositional model combining shape and appearance

  2014cited by this paper
• Very Deep Convolutional Networks for Large-Scale Image Recognition

  2014influential reference
• Adam: A Method for Stochastic Optimization

  2014cited by this paper
• Dropout: a simple way to prevent neural networks from overfitting

  2014cited by this paper
• Microsoft COCO: Common Objects in Context

  2014influential reference
• Deep visual-semantic alignments for generating image descriptions

  2014cited by this paper
• Simultaneous Detection and Segmentation

  2014cited by this paper
• Striving for Simplicity: The All Convolutional Net

  2014cited by this paper
• Deep learning in neural networks: An overview

  2014cited by this paper
• Complexity of Representation and Inference in Compositional Models with Part Sharing

  2013cited by this paper
• Learning AND-OR Templates for Object Recognition and Detection

  2013cited by this paper
• Learning with Hierarchical-Deep Models

  2013cited by this paper
• Deep Neural Networks for Object Detection

  2013cited by this paper
• Occlusion Patterns for Object Class Detection

  2013cited by this paper
• Context models and out-of-context objects

  2012cited by this paper
• Contextual Object Detection Using Set-Based Classification

  2012cited by this paper
• ImageNet classification with deep convolutional neural networks

  2012cited by this paper
• Shared parts for deformable part-based models

  2011cited by this paper
• Recognition using visual phrases

  2011cited by this paper
• Parsing Natural Scenes and Natural Language with Recursive Neural Networks

  2011cited by this paper
• Context based object categorization: A critical survey

  2010cited by this paper
• Latent hierarchical structural learning for object detection

  2010cited by this paper
• Learning the Compositional Nature of Visual Object Categories for Recognition

  2010cited by this paper
• Part and appearance sharing: Recursive Compositional Models for multi-view

  2010cited by this paper
• An empirical study of context in object detection

  2009cited by this paper
• Towards Scalable Representations of Object Categories: Learning a Hierarchy of Parts

  2007cited by this paper
• Object Recognition by Scene Alignment

  2007cited by this paper
• Author ' s personal copy The role of context in object recognition

  2007cited by this paper
• Using the Forest to See the Trees: A Graphical Model Relating Features, Objects, and Scenes

  2003cited by this paper
• GradientBased Learning Applied to Document Recognition

  2001cited by this paper
• Cortical feedback improves discrimination between figure and background by V1, V2 and V3 neurons

  1998cited by this paper
• Gradient-based learning applied to document recognition

  1998cited by this paper
• Backpropagation Applied to Handwritten Zip Code Recognition

  1989cited by this paper
• Vision: A computational investigation into the human representation and processing of visual information: David Marr. San Francisco: W. H. Freeman, 1982. pp. xvi + 397

  1983cited by this paper
• Receptive fields, binocular interaction and functional architecture in the cat's visual cortex

  1962cited by this paper
• Ieee Transactions on Pattern Analysis and Machine Intelligence Learning And-or Model to Represent Context and Occlusion for Car Detection and Viewpoint Estimation

  year unknowncited by this paper

• Let's Split Up: Zero-Shot Classifier Edits for Fine-Grained Video Understanding

  2026cites this paper
• Does Data Scaling Lead to Visual Compositional Generalization?

  2025cites this paper
• Human-like compositional learning of visually-grounded concepts using synthetic environments

  2025cites this paper
• Compositional Causal Reasoning Evaluation in Language Models

  2025cites this paper
• Enhancing Cross-Modal Medical Image Segmentation through Compositionality

  2024cites this paper
• Enhancing YOLO for occluded vehicle detection with grouped orthogonal attention and dense object repulsion

  2024cites this paper
• The Construction of Reality in an AI: A Review

  2023cites this paper
• AI-based association analysis for medical imaging using latent-space geometric confounder correction

  2023cites this paper
• Compositional Learning of Visually-Grounded Concepts Using Reinforcement

  2023cites this paper
• Compositionally Equivariant Representation Learning

  2023cites this paper
• Multi-Scale Compositional Constraints for Representation Learning on Videos

  2023cites this paper
• Human-centered deep compositional model for handling occlusions

  2023cites this paper
• Using compositionality to understand parts in whole objects

  2022cites this paper
• Compositional Generalization in Unsupervised Compositional Representation Learning: A Study on Disentanglement and Emergent Language

  2022cites this paper
• Greybox XAI: a Neural-Symbolic learning framework to produce interpretable predictions for image classification

  2022cites this paper
• Robust appearance modeling for object detection and tracking: a survey of deep learning approaches

  2022cites this paper
• Occluded Text Detection and Recognition in the Wild

  2022cites this paper
• A Model of Unified Perception and Cognition

  2022cites this paper
• Designing a Partially Understandable Neural Network through Semantic Embedding

  2021cites this paper
• Learning Compositional Representation for 4D Captures with Neural ODE

  2021cites this paper
• AGQA: A Benchmark for Compositional Spatio-Temporal Reasoning

  2021cites this paper
• Compositional Generative Networks and Robustness to Perceptible Image Changes

  2021cites this paper
• EXplainable Neural-Symbolic Learning (X-NeSyL) methodology to fuse deep learning representations with expert knowledge graphs: the MonuMAI cultural heritage use case

  2021cites this paper
• Deep Graph Convolutional Reinforcement Learning for Financial Portfolio Management - DeepPocket

  2021cites this paper
• Interpretable Compositional Convolutional Neural Networks

  2021cites this paper
• A Tutorial on Learning Disentangled Representations in the Imaging Domain

  2021cites this paper
• CX-ToM: Counterfactual explanations with theory-of-mind for enhancing human trust in image recognition models

  2021cites this paper
• Learning disentangled representations in the imaging domain

  2021cites this paper
• Semantic Explanation for Deep Neural Networks Using Feature Interactions

  2021cites this paper
• On Interpretability of Artificial Neural Networks

  2020cites this paper
• Heterogeneous Edge CNN Hardware Accelerator

  2020cites this paper
• Rule Extraction From Binary Neural Networks With Convolutional Rules for Model Validation

  2020cites this paper
• On Interpretability of Artificial Neural Networks: A Survey

  2020cites this paper
• Zero-Shot Learning from scratch (ZFS): leveraging local compositional representations

  2020influential citation
• Evaluating Compositionality of Sentence Representation Models

  2020cites this paper
• A separable neural code in monkey IT enables perfect CAPTCHA decoding

  2020cites this paper
• Interpretable and Accurate Fine-grained Recognition via Region Grouping

  2020cites this paper
• Compositional Convolutional Neural Networks: A Robust and Interpretable Model for Object Recognition Under Occlusion

  2020cites this paper
• Compositional Convolutional Neural Networks: A Deep Architecture With Innate Robustness to Partial Occlusion

  2020cites this paper
• Compositional Embeddings for Multi-Label One-Shot Learning

  2020cites this paper
• Weakly-supervised Compositional FeatureAggregation for Few-shot Recognition

  2019cites this paper
• Robustness of Object Recognition under Extreme Occlusion in Humans and Computational Models

  2019influential citation
• Do Deep Neural Networks Model Nonlinear Compositionality in the Neural Representation of Human-Object Interactions?

  2019cites this paper
• Locality and compositionality in zero-shot learning

  2019cites this paper
• Text-Guided Neural Network Training for Image Recognition in Natural Scenes and Medicine

  2019cites this paper
• Towards a Unified Evaluation of Explanation Methods without Ground Truth

  2019cites this paper
• Combining Compositional Models and Deep Networks For Robust Object Classification under Occlusion

  2019cites this paper
• Semantically Interpretable Activation Maps: what-where-how explanations within CNNs

  2019cites this paper
• X-ToM: Explaining with Theory-of-Mind for Gaining Justified Human Trust

  2019cites this paper
• Compositional Convolutional Networks For Robust Object Classification under Occlusion

  2019cites this paper
• Learning Compositional Representations for Few-Shot Recognition

  2018cites this paper
• Explaining AlphaGo: Interpreting Contextual Effects in Neural Networks

  2018cites this paper
• Interpretable Neuron Structuring with Graph Spectral Regularization

  2018cites this paper
• Unsupervised Learning of Neural Networks to Explain Neural Networks

  2018cites this paper
• Explaining Neural Networks Semantically and Quantitatively

  2018cites this paper
• Interpretable Convolutional Neural Networks

  2017cites this paper