Asymptotic Efficiency of Deterministic Estimators for Discrete Energy-Based Models: Ratio Matching and Pseudolikelihood

Standard maximum likelihood estimation cannot be applied to discrete energy-based models in the general case because the computation of exact model probabilities is intractable. Recent research has seen the proposal of several new estimators designed specifically to overcome this intractability, but virtually nothing is known about their theoretical properties. In this paper, we present a generalized estimator that unifies many of the classical and recently proposed estimators. We use results from the standard asymptotic theory for M-estimators to derive a generic expression for the asymptotic covariance matrix of our generalized estimator. We apply these results to study the relative statistical efficiency of classical pseudolikelihood and the recently-proposed ratio matching estimator.

• Publication year

  2011

• Venue

  Conference on Uncertainty in Artificial Intelligence

• Publication date

  2011-07-14

• Fields of study

  Mathematics, Computer Science

• Identifiers
  arXiv 1202.3746
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Composite Likelihood Methods

  2012cited by this paper
• Non-Local Contrastive Objectives

  2010influential reference
• Inductive Principles for Restricted Boltzmann Machine Learning

  2010cited by this paper
• Interpretation and Generalization of Score Matching

  2009influential reference
• An asymptotic analysis of generative, discriminative, and pseudolikelihood estimators

  2008cited by this paper
• Some extensions of score matching

  2007influential reference
• Contrastive Estimation: Training Log-Linear Models on Unlabeled Data

  2005influential reference
• Exponential Family Harmoniums with an Application to Information Retrieval

  2004cited by this paper
• Tree-reweighted belief propagation algorithms and approximate ML estimation by pseudo-moment matching

  2003cited by this paper
• Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data

  2001cited by this paper
• Probabilistic reasoning in intelligent systems: Networks of plausible inference

  1991cited by this paper
• Parametric Inference for imperfectly observed Gibbsian fields

  1989cited by this paper
• Information processing in dynamical systems: foundations of harmony theory

  1986cited by this paper
• A Learning Algorithm for Boltzmann Machines

  1985cited by this paper
• Markov Random Fields and Their Applications

  1980cited by this paper
• Statistical Analysis of Non-Lattice Data

  1975cited by this paper
• Beitrag zur Theorie des Ferromagnetismus

  1925cited by this paper
• On the Mathematical Foundations of Theoretical Statistics

  year unknowncited by this paper

• A survey on learning approaches for Undirected Graphical Models. Application to scene object recognition

  2017cites this paper
• Efficient machine learning using partitioned restricted Boltzmann machines

  2016cites this paper
• Efficiency Bound of Local Z-Estimators on Discrete Sample Spaces

  2016cites this paper
• Graph-based composite local Bregman divergences on discrete sample spaces

  2016cites this paper
• Classification and variable selection for high-dimensional data with applications to proteomics

  2016cites this paper
• Maximum Likelihood Learning With Arbitrary Treewidth via Fast-Mixing Parameter Sets

  2015cites this paper
• Soft-Deep Boltzmann Machines

  2015cites this paper
• Deep Boltzmann Machines with Fine Scalability

  2015cites this paper
• Analyzing Unsupervised Representation Learning Models Under the View of Dynamical Systems

  2015cites this paper
• An Empirical Investigation of Minimum Probability Flow Learning Under Different Connectivity Patterns

  2015cites this paper
• ATOMIC ENERGY MODELS FOR MACHINE LEARNING: ATOMIC

  2014cites this paper
• Understanding Minimum Probability Flow for RBMs Under Various Kinds of Dynamics

  2014cites this paper
• Distributed Parameter Estimation in Probabilistic Graphical Models

  2014cites this paper
• ATOMIC ENERGY MODELS FOR MACHINE LEARNING: ATOMIC RESTRICTED BOLTZMANN MACHINES by

  2014cites this paper
• Linear and Parallel Learning of Markov Random Fields

  2013cites this paper
• Efficient Learning of Practical Markov Random Fields with Exact Inference

  2013cites this paper
• Learning Large-Scale Conditional Random Fields

  2013cites this paper
• Unsupervised Feature Learning and Deep Learning: A Review and New Perspectives

  2012cites this paper
• Representation Learning: A Review and New Perspectives

  2012cites this paper