Hierarchical Variational Models

Black box variational inference allows researchers to easily prototype and evaluate an array of models. Recent advances allow such algorithms to scale to high dimensions. However, a central question remains: How to specify an expressive variational distribution that maintains efficient computation? To address this, we develop hierarchical variational models (HVMs). HVMs augment a variational approximation with a prior on its parameters, which allows it to capture complex structure for both discrete and continuous latent variables. The algorithm we develop is black box, can be used for any HVM, and has the same computational efficiency as the original approximation. We study HVMs on a variety of deep discrete latent variable models. HVMs generalize other expressive variational distributions and maintains higher fidelity to the posterior.

• Publication year

  2015

• Venue

  International Conference on Machine Learning

• Publication date

  2015-11-07

• Fields of study

  Mathematics, Computer Science

• Identifiers
  arXiv 1511.02386
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Graphical Models

  2020cited by this paper
• Edward: A library for probabilistic modeling, inference, and criticism

  2016cited by this paper
• Pattern Recognition And Machine Learning

  2016cited by this paper
• Auxiliary Deep Generative Models

  2016cited by this paper
• Automatic Differentiation Variational Inference

  2016cited by this paper
• Copula variational inference

  2015influential reference
• Local Expectation Gradients for Doubly Stochastic Variational Inference

  2015cited by this paper
• Variational inference with copula augmentation

  2015cited by this paper
• Variational Inference with Normalizing Flows

  2015cited by this paper
• Linear Response Methods for Accurate Covariance Estimates from Mean Field Variational Bayes

  2015cited by this paper
• Automatic Variational Inference in Stan

  2015cited by this paper
• The Variational Gaussian Process

  2015cited by this paper
• Local Expectation Gradients for Black Box Variational Inference

  2015cited by this paper
• Doubly Stochastic Variational Bayes for non-Conjugate Inference

  2014cited by this paper
• Neural Variational Inference and Learning in Belief Networks

  2014cited by this paper
• Markov Chain Monte Carlo and Variational Inference: Bridging the Gap

  2014influential reference
• Deep Exponential Families

  2014influential reference
• Structured Stochastic Variational Inference

  2014cited by this paper
• Stochastic Structured Variational Inference

  2014cited by this paper
• Stochastic Back-propagation and Variational Inference in Deep Latent Gaussian Models

  2014cited by this paper
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models

  2014cited by this paper
• Auto-Encoding Variational Bayes

  2013cited by this paper
• Learning Stochastic Inverses

  2013cited by this paper
• Black Box Variational Inference

  2013influential reference
• Fixed-Form Variational Posterior Approximation through Stochastic Linear Regression

  2012cited by this paper
• Stochastic variational inference

  2012cited by this paper
• Nonparametric variational inference

  2012cited by this paper
• Large-Scale Inference: Empirical Bayes Methods for Estimation, Testing, and Prediction

  2012cited by this paper
• A Tutorial on Bayesian Nonparametric Models

  2011cited by this paper
• Large-scale inference

  2010cited by this paper
• Presenter : HMW Category : graphical models Preference : Oral Evaluation Methods for Topic Models

  2009cited by this paper
• Data Analysis Using Regression and Multilevel/Hierarchical Models

  2009cited by this paper
• State-Dependent Exploration for Policy Gradient Methods

  2008cited by this paper
• Policy Gradients with Parameter-Based Exploration for Control

  2008cited by this paper
• Graphical Models, Exponential Families, and Variational Inference

  2008cited by this paper
• Pattern Recognition and Machine Learning

  2006cited by this paper
• An Introduction to Copulas (Springer Series in Statistics)

  2006cited by this paper
• Elements of Information Theory

  2005cited by this paper
• Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning

  2004cited by this paper
• An Auxiliary Variational Method

  2004cited by this paper
• A family of algorithms for approximate Bayesian inference

  2001cited by this paper
• Helmholtz Machines and Wake-Sleep Learning

  2000cited by this paper
• An Introduction to Copulas

  2000cited by this paper
• Propagation Algorithms for Variational Bayesian Learning

  2000cited by this paper
• Improving the Mean Field Approximation Via the Use of Mixture Distributions

  1999cited by this paper
• Learning in Graphical Models

  1999influential reference
• An Introduction to Variational Methods for Graphical Models

  1999cited by this paper
• On Structured Variational Approximations

  1997cited by this paper
• Exploiting Tractable Substructures in Intractable Networks

  1995cited by this paper
• Factorial Learning and the EM Algorithm

  1994cited by this paper
• Simple statistical gradient-following algorithms for connectionist reinforcement learning

  1992cited by this paper
• Fundamentals of statistical exponential families: with applications in statistical decision theory

  1986cited by this paper
• Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper

  1977cited by this paper
• Combining Possibly Related Estimation Problems

  1973cited by this paper
• The Empirical Bayes Approach to Statistical Decision Problems

  1964cited by this paper
• A Stochastic Approximation Method

  1951cited by this paper

• Disentangling consistent, complementary and modality-private information via twin information bottleneck for multi-modal clustering

  2026cites this paper
• Discrete diffusion samplers and bridges: Off-policy algorithms and applications in latent spaces

  2026cites this paper
• A Structurally Localized Ensemble Kalman Filtering Approach

  2026cites this paper
• A Kernel Approach for Semi-implicit Variational Inference

  2026cites this paper
• Reinforced sequential Monte Carlo for amortised sampling

  2025influential citation
• Weighted Fisher divergence for high-dimensional Gaussian variational inference

  2025cites this paper
• Asymptotically exact variational flows via involutive MCMC kernels

  2025cites this paper
• Variational Inference Aided Variable Selection For Spatially Structured High Dimensional Covariates

  2025cites this paper
• Demystifying Variational Diffusion Models

  2025cites this paper
• Continuous Semi-Implicit Models

  2025cites this paper
• Neural Stochastic Flows: Solver-Free Modelling and Inference for SDE Solutions

  2025cites this paper
• Bend to Mend: Toward Trustworthy Variational Bayes with Valid Uncertainty Quantification

  2025cites this paper
• Improving Non-Autoregressive Response Generation by Distilling Posterior Information

  2025cites this paper
• A Hierarchical Variational Graph Fused Lasso for Recovering Relative Rates in Spatial Compositional Data

  2025cites this paper
• Probabilistic Digital Twins of Users: Latent Representation Learning with Statistically Validated Semantics

  2025cites this paper
• Beyond Verifiable Rewards: Scaling Reinforcement Learning for Language Models to Unverifiable Data

  2025cites this paper
• The Use of Variational Inference for Lifetime Data with Spatial Correlations

  2025cites this paper
• Left Barrier Loss for Unbiased Survival Analysis Prediction

  2025cites this paper
• Personalized design aesthetic preference modeling: a variational autoencoder and meta-learning approach for multi-modal feature representation and transfer optimization

  2025cites this paper
• On Context-Content Uncertainty Principle

  2025cites this paper
• DIME:Diffusion-Based Maximum Entropy Reinforcement Learning

  2025cites this paper
• Contrasting with Symile: Simple Model-Agnostic Representation Learning for Unlimited Modalities

  2024cites this paper
• Semi-Airborne Transient Electromagnetic Denoising Through Variation Diffusion Model

  2024cites this paper
• On the Importance of the Execution Schedule for Bayesian Inference

  2024cites this paper
• Scene-Dependent Prediction in Latent Space for Video Anomaly Detection and Anticipation

  2024cites this paper
• Integrating Random Effects in Variational Autoencoders for Dimensionality Reduction of Correlated Data

  2024cites this paper
• ELBOing Stein: Variational Bayes with Stein Mixture Inference

  2024cites this paper
• What's the score? Automated Denoising Score Matching for Nonlinear Diffusions

  2024cites this paper
• Denoising Diffusion Variational Inference: Diffusion Models as Expressive Variational Posteriors

  2024cites this paper
• On Neural Process Pooling Operator Choice and Posterior Calibration

  2024cites this paper
• Hierarchical VAE with a Diffusion-based VampPrior

  2024cites this paper
• A Review of Bayesian Uncertainty Quantification in Deep Probabilistic Image Segmentation

  2024cites this paper
• Unified Cross-Modal Medical Image Synthesis With Hierarchical Mixture of Product-of-Experts

  2024cites this paper
• Open-Vocabulary Predictive World Models from Sensor Observations

  2024cites this paper
• Mathematical Engineering of Deep Learning

  2024cites this paper
• Combining Hierachical VAEs with LLMs for clinically meaningful timeline summarisation in social media

  2024cites this paper
• A Survey on Variational Autoencoders in Recommender Systems

  2024cites this paper
• Benefits of Non-Linear Scale Parameterizations in Black Box Variational Inference through Smoothness Results and Gradient Variance Bounds

  2024cites this paper
• Integrative Variational Autoencoders for Generative Modeling of an Image Outcome with Multiple Input Images

  2024cites this paper
• A Probabilistic Model behind Self- Supervised Learning

  2024cites this paper
• Federated Learning for Non-factorizable Models using Deep Generative Prior Approximations

  2024cites this paper
• Provably Scalable Black-Box Variational Inference with Structured Variational Families

  2024cites this paper
• Stochastic reconstruction of digital cores using two-discriminator VAE-GAN

  2024cites this paper
• Probabilistic Programming with Programmable Variational Inference

  2024cites this paper
• Scalable Vertical Federated Learning via Data Augmentation and Amortized Inference

  2024cites this paper
• Calibrating the Rigged Lottery: Making All Tickets Reliable

  2023cites this paper
• Collective relational inference for learning heterogeneous interactions

  2023cites this paper
• Where to Diffuse, How to Diffuse, and How to Get Back: Automated Learning for Multivariate Diffusions

  2023cites this paper
• Multinomial belief networks for healthcare data

  2023cites this paper
• Learning Energy-Based Prior Model with Diffusion-Amortized MCMC

  2023cites this paper
• RPIFL: Reliable and Privacy-Preserving Federated Learning for the Internet of Things

  2023cites this paper
• Hierarchical Graph Neural Network Based on Semi-Implicit Variational Inference

  2023cites this paper
• Tree Variational Autoencoders

  2023cites this paper
• Process Monitoring Using Domain-Adversarial Probabilistic Principal Component Analysis: A Transfer Learning Framework

  2023cites this paper
• Local Search GFlowNets

  2023cites this paper
• Learning variational autoencoders via MCMC speed measures

  2023cites this paper
• Self-Organizing a Latent Hierarchy of Sketch Patterns for Controllable Sketch Synthesis

  2023cites this paper
• Learning Directed Graphical Models with Optimal Transport

  2023cites this paper
• Parameter Estimation in DAGs from Incomplete Data via Optimal Transport

  2023cites this paper
• Unified Brain MR-Ultrasound Synthesis using Multi-Modal Hierarchical Representations

  2023cites this paper
• Diffusion models for probabilistic programming

  2023cites this paper
• Hierarchical Semi-Implicit Variational Inference with Application to Diffusion Model Acceleration

  2023cites this paper
• Learning to Jump: Thinning and Thickening Latent Counts for Generative Modeling

  2023cites this paper
• Collective Relational Inference for learning physics-consistent heterogeneous particle interactions

  2023cites this paper
• Variational Inference for Bayesian Neural Networks under Model and Parameter Uncertainty

  2023cites this paper
• Predictive World Models from Real-World Partial Observations

  2023cites this paper
• Human Activity Recognition with an HMM-Based Generative Model

  2023cites this paper
• Sparsifying Bayesian neural networks with latent binary variables and normalizing flows

  2023cites this paper
• AdaVAE: Bayesian Structural Adaptation for Variational Autoencoders

  2023cites this paper
• Order-Preserving GFlowNets

  2023cites this paper
• Deep Hierarchical Variational Autoencoders for World Models in Reinforcement Learning

  2023cites this paper
• Variational Nonlinear Kalman Filtering With Unknown Process Noise Covariance

  2023cites this paper
• Multinomial belief networks

  2023cites this paper
• Constraining cosmological parameters from N-body simulations with variational Bayesian neural networks

  2023cites this paper
• 3D Variational Bayesian Full Waveform Inversion

  2022cites this paper
• 3D Bayesian Variational Full Waveform Inversion

  2022cites this paper
• GFlowNets and variational inference

  2022influential citation
• Large-Scale Auto-Regressive Modeling Of Street Networks

  2022cites this paper
• Unifying Generative Models with GFlowNets

  2022cites this paper
• Alleviating Adversarial Attacks on Variational Autoencoders with MCMC

  2022cites this paper
• Amortized Variational Inference: A Systematic Review

  2022cites this paper
• Latent Variable Modelling Using Variational Autoencoders: A survey

  2022influential citation
• Quasi-site-specific soil property prediction using a cluster-based hierarchical Bayesian model

  2022cites this paper
• VFDS: Variational Foresight Dynamic Selection in Bayesian Neural Networks for Efficient Human Activity Recognition

  2022cites this paper
• Higher-Order Generalization Bounds: Learning Deep Probabilistic Programs via PAC-Bayes Objectives

  2022cites this paper
• Break The Spell Of Total Correlation In betaTCVAE

  2022cites this paper
• Defending Variational Autoencoders from Adversarial Attacks with MCMC

  2022cites this paper
• Efficient-VDVAE: Less is more

  2022cites this paper
• Ergodic variational flows

  2022cites this paper
• Amortized Variational Inference: Towards the Mathematical Foundation and Review

  2022cites this paper
• Rethinking Learning Dynamics in RL using Adversarial Networks

  2022cites this paper
• Unsupervised Bayesian classification for models with scalar and functional covariates.

  2022cites this paper
• Deep Generative Models in the Industrial Internet of Things: A Survey

  2022cites this paper
• Multicategory choice modeling with sparse and high dimensional data: A Bayesian deep learning approach

  2022cites this paper
• MixFlows: principled variational inference via mixed flows

  2022cites this paper
• An Approximated Collapsed Variational Bayes Approach to Variable Selection in Linear Regression

  2022cites this paper
• Break The Spell Of Total Correlation In β −TCVAE

  2022cites this paper
• Black-box Coreset Variational Inference

  2022cites this paper
• Moment Propagation

  2022cites this paper
• β-Multivariational Autoencoder for Entangled Representation Learning in Video Frames

  2022cites this paper