Deep Value Networks Learn to Evaluate and Iteratively Refine Structured Outputs

We approach structured output prediction by optimizing a deep value network (DVN) to precisely estimate the task loss on different output configurations for a given input. Once the model is trained, we perform inference by gradient descent on the continuous relaxations of the output variables to find outputs with promising scores from the value network. When applied to image segmentation, the value network takes an image and a segmentation mask as inputs and predicts a scalar estimating the intersection over union between the input and ground truth masks. For multi-label classification, the DVN's objective is to correctly predict the F1 score for any potential label configuration. The DVN framework achieves the state-of-the-art results on multi-label prediction and image segmentation benchmarks.

• Publication year

  2017

• Venue

  International Conference on Machine Learning

• Publication date

  2017-03-13

• Fields of study

  Computer Science

• Identifiers
  arXiv 1703.04363
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• End-to-End Learning for Structured Prediction Energy Networks

  2017cited by this paper
• Pixel Recursive Super Resolution

  2017cited by this paper
• Conditional Image Generation with PixelCNN Decoders

  2016cited by this paper
• DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs

  2016cited by this paper
• Reward Augmented Maximum Likelihood for Neural Structured Prediction

  2016cited by this paper
• Input Convex Neural Networks

  2016cited by this paper
• Perceptual Losses for Real-Time Style Transfer and Super-Resolution

  2016cited by this paper
• Learning to Refine Object Segments

  2016cited by this paper
• WaveNet: A Generative Model for Raw Audio

  2016cited by this paper
• Synthesizing the preferred inputs for neurons in neural networks via deep generator networks

  2016cited by this paper
• A Learned Representation For Artistic Style

  2016cited by this paper
• Heterogeneous substitution systems revisited

  2016cited by this paper
• Deep Learning for Semantic Part Segmentation with High-Level Guidance

  2015influential reference
• Learning Deconvolution Network for Semantic Segmentation

  2015cited by this paper
• Iterative Instance Segmentation

  2015cited by this paper
• Conditional Random Fields as Recurrent Neural Networks

  2015cited by this paper
• Inceptionism: Going Deeper into Neural Networks

  2015cited by this paper
• Learning shape priors for object segmentation via neural networks

  2015influential reference
• U-Net: Convolutional Networks for Biomedical Image Segmentation

  2015cited by this paper
• Training Deep Neural Networks via Direct Loss Minimization

  2015cited by this paper
• Deep Reinforcement Learning with Double Q-Learning

  2015cited by this paper
• Structured Prediction Energy Networks

  2015influential reference
• A Neural Algorithm of Artistic Style

  2015cited by this paper
• Object segmentation with deep regression

  2015cited by this paper
• Fully convolutional networks for semantic segmentation

  2014influential reference
• Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-scale Convolutional Architecture

  2014cited by this paper
• Explaining and Harnessing Adversarial Examples

  2014cited by this paper
• Max-Margin Boltzmann Machines for Object Segmentation

  2014influential reference
• Sequence to Sequence Learning with Neural Networks

  2014cited by this paper
• Learning Deep Structured Models

  2014cited by this paper
• Neural Machine Translation by Jointly Learning to Align and Translate

  2014cited by this paper
• Hypercolumns for object segmentation and fine-grained localization

  2014cited by this paper
• Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs

  2014cited by this paper
• Multi-Label Learning with Posterior Regularization

  2014influential reference
• Augmenting CRFs with Boltzmann Machine Shape Priors for Image Labeling

  2013influential reference
• Exploring Compositional High Order Pattern Potentials for Structured Output Learning

  2013influential reference
• Intriguing properties of neural networks

  2013cited by this paper
• Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation

  2013cited by this paper
• Semantic segmentation using regions and parts

  2012cited by this paper
• The Shape Boltzmann Machine: A Strong Model of Object Shape

  2012cited by this paper
• Semantic Segmentation with Second-Order Pooling

  2012cited by this paper
• ImageNet classification with deep convolutional neural networks

  2012cited by this paper
• Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials

  2011cited by this paper
• ImageNet: A large-scale hierarchical image database

  2009cited by this paper
• Multilabel Text Classification for Automated Tag Suggestion

  2008cited by this paper
• Robust Higher Order Potentials for Enforcing Label Consistency

  2008cited by this paper
• Labeled Faces in the Wild: A Database forStudying Face Recognition in Unconstrained Environments

  2008cited by this paper
• A Tutorial on Energy-Based Learning

  2006cited by this paper
• Support vector machine learning for interdependent and structured output spaces

  2004influential reference
• Learning to Segment

  2004cited by this paper
• Max-Margin Markov Networks

  2003influential reference
• Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data

  2001cited by this paper
• Fast approximate energy minimization via graph cuts

  2001cited by this paper
• Loopy Belief Propagation for Approximate Inference: An Empirical Study

  1999influential reference
• Reinforcement Learning: An Introduction

  1998cited by this paper
• Learning from delayed rewards

  1995cited by this paper
• Q-learning

  1992cited by this paper
• Noname manuscript No. (will be inserted by the editor) Inference Methods for CRFs with Co-occurrence Statistics

  year unknowncited by this paper

• Deep Sketched Output Kernel Regression for Structured Prediction

  2024cites this paper
• Structured Prediction with Stronger Consistency Guarantees

  2023cites this paper
• Implicit Training of Inference Network Models for Structured Prediction

  2023cites this paper
• Energy-Based Models for Cross-Modal Localization using Convolutional Transformers

  2023cites this paper
• Interval Type-2 Fuzzy Neural Networks for Multi-Label Classification

  2023cites this paper
• Sketch In, Sketch Out: Accelerating both Learning and Inference for Structured Prediction with Kernels

  2023cites this paper
• Generalized Zero-Shot Activity Recognition with Embedding-Based Method

  2023cites this paper
• Adversarial structured prediction for domain-adaptive semantic segmentation

  2022cites this paper
• Implicit Training of Energy Model for Structure Prediction

  2022influential citation
• SOInter: A Novel Deep Energy Based Interpretation Method for Explaining Structured Output Models

  2022influential citation
• Vector-Valued Least-Squares Regression under Output Regularity Assumptions

  2022influential citation
• Neuro-Symbolic Constraint Programming for Structured Prediction

  2021cites this paper
• N O C ONDITIONAL M ODELS FOR ME : T RAINING J OINT EBM S ON M IXED C ONTINUOUS AND D ISCRETE D ATA

  2021cites this paper
• Flow-Based Spatio-Temporal Structured Prediction of Motion Dynamics

  2021cites this paper
• Structured Prediction in NLP - A survey

  2021cites this paper
• Directly Training Joint Energy-Based Models for Conditional Synthesis and Calibrated Prediction of Multi-Attribute Data

  2021cites this paper
• Structured Convolutional Kernel Networks for Airline Crew Scheduling

  2021cites this paper
• Optimizing Non-Differentiable Metrics for Hashing

  2021cites this paper
• Efficient End-to-end Learning of Cross-event Dependencies for Document-level Event Extraction

  2020influential citation
• Differentiable Fixed-Point Iteration Layer

  2020cites this paper
• Adversarial Localized Energy Network for Structured Prediction

  2020influential citation
• Decoding As Dynamic Programming For Recurrent Autoregressive Models

  2020cites this paper
• A Perspective on Deep Learning for Molecular Modeling and Simulations

  2020cites this paper
• Differentiable Forward and Backward Fixed-Point Iteration Layers

  2020cites this paper
• Learning Output Embeddings in Structured Prediction

  2020cites this paper
• Task-Aware Performance Prediction for Efficient Architecture Search

  2020cites this paper
• Towards Structured Prediction in Bioinformatics with Deep Learning

  2020cites this paper
• F-Measure Optimisation and Label Regularisation for Energy-Based Neural Dialogue State Tracking Models

  2020influential citation
• Energy-based Neural Modelling for Large-Scale Multiple Domain Dialogue State Tracking

  2020cites this paper
• Energy-Based Models for Continual Learning

  2020cites this paper
• Document-level Event Extraction with Efficient End-to-end Learning of Cross-event Dependencies

  2020cites this paper
• Investigating Variable Dependencies in Dialogue States

  2019cites this paper
• Feature-Critic Networks for Heterogeneous Domain Generalization

  2019cites this paper
• The 2nd Learning from Limited Labeled Data (LLD) Workshop: Representation Learning for Weak Supervision and Beyond

  2019cites this paper
• Neural Message Passing for Multi-Label Classification

  2019cites this paper
• Deep Reinforcement Learning Meets Structured Prediction

  2019cites this paper
• Fast Task-Aware Architecture Inference

  2019cites this paper
• Learning Assistance from an Adversarial Critic for Multi-Outputs Prediction

  2019influential citation
• Learning and Inference for Structured Prediction: A Unifying Perspective

  2019cites this paper
• Dynamic Scale Inference by Entropy Minimization

  2019cites this paper
• Optimizing Through Learned Errors for Accurate Sports Field Registration

  2019cites this paper
• Graph Structured Prediction Energy Networks

  2019cites this paper
• Capturing Dialogue State Variable Dependencies with an Energy-based Neural Dialogue State Tracker

  2019cites this paper
• Adversarial Loss as Prior Regularization for Structured Prediction

  2019influential citation
• Towards Learning Structure via Consensus for Face Segmentation and Parsing

  2019cites this paper
• Learning to Calibrate and Rerank Multi-label Predictions

  2019influential citation
• AN ABSTRACT OF THE THESIS OF Chao Ma for the degree of Doctor of Philosophy in Computer Science presented on July 15, 2019. Title: New Directions in Search-based Structured Prediction: Multi-Task Learning and Integration of Deep Models Abstract approved:

  2019influential citation
• Continuous Adaptation for Interactive Object Segmentation by Learning from Corrections

  2019cites this paper
• Grid-Based Micro Traffic Prediction using Fully Convolutional Networks

  2019cites this paper
• Graph Structured Prediction Energy Net Algorithms

  2019cites this paper
• Randomized Greedy Search for Structured Prediction: Amortized Inference and Learning

  2019influential citation
• Structured Output Learning with Conditional Generative Flows

  2019cites this paper
• Predict and Constrain: Modeling Cardinality in Deep Structured Prediction

  2018influential citation
• Mapping Images to Scene Graphs with Permutation-Invariant Structured Prediction

  2018influential citation
• Learning Discriminators as Energy Networks in Adversarial Learning

  2018influential citation
• Adversarial Structure Matching Loss for Image Segmentation

  2018cites this paper
• Deep Structured Prediction with Nonlinear Output Transformations

  2018cites this paper
• Non-parametric Bayesian models for structured output prediction

  2018cites this paper
• Training Structured Prediction Energy Networks with Indirect Supervision

  2018influential citation
• Search-Guided, Lightly-supervised Training of Structured Prediction Energy Networks

  2018influential citation
• Adversarial Structure Matching for Structured Prediction Tasks

  2018cites this paper
• Deep Energy-Based Models for Structured Prediction

  2017cites this paper
• End-to-End Learning for Structured Prediction Energy Networks

  2017cites this paper
• Interest-Based Video Summarization via Subset Selection

  2017cites this paper
• The 2nd Learning from Limited Labeled Data (LLD) Workshop: Representation Learning for Weak Supervision and Beyond

  year unknowncites this paper
• The 2nd Learning from Limited Labeled Data (LLD) Workshop: Representation Learning for Weak Supervision and Beyond

  year unknowncites this paper
• Structured Energy Network as a Loss Function

  year unknowninfluential citation
• Projections Aléatoires Entrée, Sortie : Accélération de l’Apprentissage et de l’Inférence dans la Prédiction Structurée avec Noyaux

  year unknowncites this paper