Efficient inference in occlusion-aware generative models of images

We present a generative model of images based on layering, in which image layers are individually generated, then composited from front to back. We are thus able to factor the appearance of an image into the appearance of individual objects within the image --- and additionally for each individual object, we can factor content from pose. Unlike prior work on layered models, we learn a shape prior for each object/layer, allowing the model to tease out which object is in front by looking for a consistent shape, without needing access to motion cues or any labeled data. We show that ordinary stochastic gradient variational bayes (SGVB), which optimizes our fully differentiable lower-bound on the log-likelihood, is sufficient to learn an interpretable representation of images. Finally we present experiments demonstrating the effectiveness of the model for inferring foreground and background objects in images.

• Publication year

  2015

• Venue

  arXiv.org

• Publication date

  2015-11-19

• Fields of study

  Mathematics, Computer Science

• Identifiers
  arXiv 1511.06362
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Generative Adversarial Networks

  2021cited by this paper
• Under Review as a Conference Paper at Iclr 2017 Delving into Transferable Adversarial Ex- Amples and Black-box Attacks

  2016cited by this paper
• Deep Convolutional Inverse Graphics Network

  2015cited by this paper
• Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks

  2015cited by this paper
• Importance Weighted Autoencoders

  2015cited by this paper
• DRAW: A Recurrent Neural Network For Image Generation

  2015influential reference
• Amodal Completion and Size Constancy in Natural Scenes

  2015cited by this paper
• Spatial Transformer Networks

  2015influential reference
• Semi-supervised Learning with Deep Generative Models

  2014cited by this paper
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models

  2014cited by this paper
• Discovering Hidden Factors of Variation in Deep Networks

  2014cited by this paper
• Learning to generate chairs with convolutional neural networks

  2014cited by this paper
• Simultaneous Detection and Segmentation

  2014cited by this paper
• Scene Collaging: Analysis and Synthesis of Natural Images with Semantic Layers

  2013cited by this paper
• Auto-Encoding Variational Bayes

  2013cited by this paper
• Layered Object Models for Image Segmentation

  2012cited by this paper
• Learning a Generative Model of Images by Factoring Appearance and Shape

  2011cited by this paper
• Transforming Auto-Encoders

  2011cited by this paper
• Understanding the difficulty of training deep feedforward neural networks

  2010cited by this paper
• Fast Transformation-Invariant Component Analysis

  2008cited by this paper
• Graphical Models, Exponential Families, and Variational Inference

  2008cited by this paper
• Generative Model for Layers of Appearance and Deformation

  2005cited by this paper
• Greedy Learning of Multiple Objects in Images Using Robust Statistics and Factorial Learning

  2004cited by this paper
• View from the Top: Hierarchies and Reverse Hierarchies in the Visual System

  2002cited by this paper
• Gradient-based learning applied to document recognition

  1998cited by this paper
• Layered representation of motion video using robust maximum-likelihood estimation of mixture models and MDL encoding

  1995cited by this paper
• Representing moving images with layers

  1994cited by this paper
• Robust estimation of a multi-layered motion representation

  1991cited by this paper
• Compositing digital images

  1984cited by this paper
• Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper

  1977cited by this paper

• Unsupervised Object Discovery: A Comprehensive Survey and Unified Taxonomy

  2024influential citation
• Unsupervised Object-Centric Learning From Multiple Unspecified Viewpoints

  2024cites this paper
• Small object detection in diverse application landscapes: a survey

  2024cites this paper
• Enhanced Compressed Sensing Based Deep Learning Neural Network for Single Image Super Resolution of COVID-19 Using X-Ray Images

  2023cites this paper
• Benchmarking and Analysis of Unsupervised Object Segmentation from Real-World Single Images

  2023cites this paper
• COAT: Measuring Object Compositionality in Emergent Representations

  2022cites this paper
• Object-wise Masked Autoencoders for Fast Pre-training

  2022cites this paper
• Promising or Elusive? Unsupervised Object Segmentation from Real-world Single Images

  2022cites this paper
• Neuromorphic visual scene understanding with resonator networks

  2022cites this paper
• Compositional Scene Representation Learning via Reconstruction: A Survey

  2022cites this paper
• APEX: Unsupervised, Object-Centric Scene Segmentation and Tracking for Robot Manipulation

  2021cites this paper
• MarioNette: Self-Supervised Sprite Learning

  2021cites this paper
• GENESIS-V2: Inferring Unordered Object Representations without Iterative Refinement

  2021influential citation
• Knowledge-Guided Object Discovery with Acquired Deep Impressions

  2021cites this paper
• Acknowledgments and Disclosure of Funding

  2021cites this paper
• Unsupervised Learning of Compositional Scene Representations from Multiple Unspecified Viewpoints

  2021cites this paper
• Cyclic Diffeomorphic Transformer Nets For Contour Alignment

  2021influential citation
• Reconstruction Bottlenecks in Object-Centric Generative Models

  2020cites this paper
• Unsupervised part representation by Flow Capsules

  2020cites this paper
• DRCS-SR: Deep Robust Compressed Sensing for Single Image Super-Resolution

  2020cites this paper
• Physics-as-Inverse-Graphics: Joint Unsupervised Learning of Objects and Physics from Video

  2019cites this paper
• Unsupervised Discovery of Parts, Structure, and Dynamics

  2019cites this paper
• Context-Aware Automatic Occlusion Removal

  2019cites this paper
• Generative Modeling of Infinite Occluded Objects for Compositional Scene Representation

  2019cites this paper
• GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations

  2019cites this paper
• Priming Deep Pedestrian Detection with Geometric Context

  2019cites this paper
• Robust Image Completion via Deep Feature Transformations

  2019cites this paper
• Physics-as-Inverse-Graphics: Unsupervised Physical Parameter Estimation from Video

  2019cites this paper
• Learning to Exploit Stability for 3D Scene Parsing

  2018cites this paper
• Unsupervised Multi-Object Detection for Video Surveillance Using Memory-Based Recurrent Attention Networks

  2018cites this paper
• Learning Segmentation Masks with the Independence Prior

  2018cites this paper
• Learning Single-view 3D Reconstruction of Objects and Scenes

  2018cites this paper
• Deep Diffeomorphic Transformer Networks

  2018cites this paper
• Tracking by Animation: Unsupervised Learning of Multi-Object Attentive Trackers

  2018cites this paper
• Learning to See Physics via Visual De-animation

  2017cites this paper
• Neural Scene De-rendering

  2017cites this paper
• Factorized Variational Autoencoders for Modeling Audience Reactions to Movies

  2017cites this paper
• Tagger: Deep Unsupervised Perceptual Grouping

  2016cites this paper
• Deep Successor Reinforcement Learning

  2016cites this paper
• Attend, Infer, Repeat: Fast Scene Understanding with Generative Models

  2016cites this paper
• One-Shot Generalization in Deep Generative Models

  2016influential citation
• Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation

  2016cites this paper
• Learning Shape Abstractions by Assembling Volumetric Primitives

  2016cites this paper
• LR-GAN: Layered Recursive Generative Adversarial Networks for Image Generation

  2016cites this paper