Variational Cumulant Expansions for Intractable Distributions

Intractable distributions present a common difficulty in inference within the probabilistic knowledge representation framework and variational methods have recently been popular in providing an approximate solution. In this article, we describe a perturbational approach in the form of a cumulant expansion which, to lowest order, recovers the standdard Kullback-Leibler variational bound. Higher-order terms describe corrections on the variational approach without incurring much further computational cost. The relationship to other perturbational approaches such as TAP is also elucidated. We demonstrate the method on a particular class of undirected graphical models, Boltzmann machines, for which our simulation results confirm improved accuracy and enhanced stability during learning.

• Publication year

  2011

• Venue

  Journal of Artificial Intelligence Research

• Publication date

  2011-05-26

• Fields of study

  Mathematics, Computer Science

• Identifiers
  DOI 10.1613/jair.567arXiv 1105.5455
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Improving the Mean Field Approximation Via the Use of Mixture Distributions

  1999cited by this paper
• Pruning Using Parameter and Neuronal Metrics

  1999cited by this paper
• Tractable Undirected Approximations for Graphical Models

  1998cited by this paper
• Mean Field Theory based on Belief Networks for Approximate Inference

  1998influential reference
• Ensemble learning in Bayesian neural networks

  1998cited by this paper
• Mixture Representations for Inference and Learning in Boltzmann Machines

  1998cited by this paper
• Efficient Learning in Boltzmann Machines Using Linear Response Theory

  1998cited by this paper
• Tractable Variational Structures for Approximating Graphical Models

  1998influential reference
• Variational Methods for Inference and Estimation inGraphical

  1997influential reference
• Variational methods for inference and estimation in graphical models

  1997influential reference
• Bericht 97-5 Des Fachbereichs Informatik Der Technischen Universitt at Berlin Eecient Inference and Learning in Decimatable Boltzmann Machines

  1997cited by this paper
• Boltzmann Machine Learning Using Mean Field Theory and Linear Response Correction

  1997cited by this paper
• Mean Field Theory for Sigmoid Belief Networks

  1996influential reference
• Bucket elimination: A unifying framework for probabilistic inference

  1996cited by this paper
• Probable networks and plausible predictions - a review of practical Bayesian methods for supervised neural networks

  1995cited by this paper
• A new method to optimize the satellite broadcasting schedules using the mean field annealing of a Hopfield neural network

  1995cited by this paper
• Learning in Boltzmann Trees

  1994cited by this paper
• Neural Networks for Pattern Recognition

  1993cited by this paper
• The limitations of deterministic Boltzmann machine learning

  1993cited by this paper
• Learning processes in neural networks.

  1991cited by this paper
• The Computational Complexity of Probabilistic Inference Using Bayesian Belief Networks

  1990cited by this paper
• Probabilistic reasoning in intelligent systems - networks of plausible inference

  1989cited by this paper
• Probabilistic reasoning in intelligent systems

  1988cited by this paper
• A Mean Field Theory Learning Algorithm for Neural Networks

  1987influential reference
• A Learning Algorithm for Boltzmann Machines

  1985cited by this paper
• TAP free energy structure of SK spin glasses

  1985cited by this paper
• Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images

  1984cited by this paper
• Convergence condition of the TAP equation for the infinite-ranged Ising spin glass model

  1982cited by this paper
• Evidence for massless modes in the 'solvable model' of a spin glass

  1979cited by this paper
• Solution of 'Solvable model of a spin glass'

  1977cited by this paper

• Probing Geometry of Next Token Prediction Using Cumulant Expansion of the Softmax Entropy

  2025cites this paper
• A sparse expansion for deep Gaussian processes

  2021cites this paper
• Bayesian Brains and the Rényi Divergence

  2021cites this paper
• Variational Inference: A Review for Statisticians

  2016cites this paper
• Collective behaviours in the stock market: a maximum entropy approach

  2014cites this paper
• N ov 2 01 2 Market structure explained by pairwise interactions

  2012cites this paper
• Market structure explained by pairwise interactions

  2012cites this paper
• Statistical pairwise interaction model of stock market

  2012cites this paper
• Irregular-structure tree models for image interpretation

  2005cites this paper
• On the Concentration of Expectation and Approximate Inference in Layered Networks

  2003cites this paper
• Stochastic processes on graphs with cycles: geometric and variational approaches

  2002cites this paper
• Mean Field Methods for a Special Class of Belief Networks

  2001cites this paper
• Cumulant Expansions of Reduced Densities, Reduced Density Matrices, and Green’s Functions

  2000cites this paper
• Approximate Inference A lgorithms for Two-Layer Bayesian Networks

  1999cites this paper
• Tractable Variational Structures for Approximating Graphical Models

  1998cites this paper