SCIENCE@home

Implicit Density Estimation by Local Moment Matching to Sample from Auto-Encoders

Yoshua Bengio,Guillaume Alain,Salah Rifai

Published 2012 in arXiv.org

ABSTRACT

Recent work suggests that some auto-encoder variants do a good job of capturing the local manifold structure of the unknown data generating density. This paper contributes to the mathematical understanding of this phenomenon and helps define better justified sampling algorithms for deep learning based on auto-encoder variants. We consider an MCMC where each step samples from a Gaussian whose mean and covariance matrix depend on the previous state, defines through its asymptotic distribution a target density. First, we show that good choices (in the sense of consistency) for these mean and covariance functions are the local expected value and local covariance under that target density. Then we show that an auto-encoder with a contractive penalty captures estimators of these local moments in its reconstruction function and its Jacobian. A contribution of this work is thus a novel alternative to maximum-likelihood density estimation, which we call local moment matching. It also justifies a recently proposed sampling algorithm for the Contractive Auto-Encoder and extends it to the Denoising Auto-Encoder.

PUBLICATION RECORD

  • Publication year

    2012

  • Venue

    arXiv.org

  • Publication date

    2012-06-30

  • Fields of study

    Mathematics, Computer Science

  • Identifiers
    arXiv 1207.0057
  • External record

    Open on Semantic Scholar

  • Source metadata

    Semantic Scholar

CITATION MAP

EXTRACTION MAP

CLAIMS

  • No claims are published for this paper.

CONCEPTS

  • No concepts are published for this paper.

REFERENCES

Showing 1-30 of 30 references · Page 1 of 1

CITED BY

Showing 1-13 of 13 citing papers · Page 1 of 1