What regularized auto-encoders learn from the data-generating distribution

What do auto-encoders learn about the underlying data generating distribution? Recent work suggests that some auto-encoder variants do a good job of capturing the local manifold structure of data. This paper clarifies some of these previous observations by showing that minimizing a particular form of regularized reconstruction error yields a reconstruction function that locally characterizes the shape of the data generating density. We show that the auto-encoder captures the score (derivative of the log-density with respect to the input). It contradicts previous interpretations of reconstruction error as an energy function. Unlike previous results, the theorems provided here are completely generic and do not depend on the parametrization of the auto-encoder: they show what the auto-encoder would tend to if given enough capacity and examples. These results are for a contractive training criterion we show to be similar to the denoising auto-encoder training criterion with small corruption noise, but with contraction applied on the whole reconstruction function rather than just encoder. Similarly to score matching, one can consider the proposed training criterion as a convenient alternative to maximum likelihood because it does not involve a partition function. Finally, we show how an approximate Metropolis-Hastings MCMC can be setup to recover samples from the estimated distribution, and this is confirmed in sampling experiments.

• Publication year

  2012

• Venue

  Journal of machine learning research

• Publication date

  2012-11-18

• Fields of study

  Mathematics, Computer Science

• Identifiers
  DOI 10.5555/2627435.2750359arXiv 1211.4246
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Generalized Denoising Auto-Encoders as Generative Models

  2013cited by this paper
• Representation Learning: A Review and New Perspectives

  2012influential reference
• A Generative Process for Contractive Auto-Encoders

  2012cited by this paper
• Implicit Density Estimation by Local Moment Matching to Sample from Auto-Encoders

  2012influential reference
• Better Mixing via Deep Representations

  2012cited by this paper
• Unsupervised Feature Learning and Deep Learning: A Review and New Perspectives

  2012influential reference
• A Connection Between Score Matching and Denoising Autoencoders

  2011influential reference
• On the Expressive Power of Deep Architectures

  2011cited by this paper
• Structured sparse coding via lateral inhibition

  2011cited by this paper
• The Manifold Tangent Classifier

  2011influential reference
• On Autoencoders and Score Matching for Energy Based Models

  2011cited by this paper
• Contractive Auto-Encoders: Explicit Invariance During Feature Extraction

  2011influential reference
• Sample Complexity of Testing the Manifold Hypothesis

  2010cited by this paper
• Regularized estimation of image statistics by Score Matching

  2010cited by this paper
• Learning invariant features through topographic filter maps

  2009cited by this paper
• Deep Boltzmann Machines

  2009cited by this paper
• Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations

  2009cited by this paper
• Extracting and composing robust features with denoising autoencoders

  2008influential reference
• Natural Image Denoising with Convolutional Networks

  2008cited by this paper
• Introduction to the Calculus of Variations

  2008cited by this paper
• Some extensions of score matching

  2007cited by this paper
• Learning Deep Architectures for AI

  2007cited by this paper
• Sparse Feature Learning for Deep Belief Networks

  2007cited by this paper
• Efficient Learning of Sparse Representations with an Energy-Based Model

  2006cited by this paper
• A Fast Learning Algorithm for Deep Belief Nets

  2006cited by this paper
• Estimation of Non-Normalized Statistical Models by Score Matching

  2005influential reference
• Algorithms for manifold learning

  2005cited by this paper
• Monte Carlo Statistical Methods

  2005cited by this paper
• Introduction to the Calculus of Variations

  1999cited by this paper
• Sparse coding with an overcomplete basis set: a strategy employed by V1?

  1997cited by this paper

• GEPC: Group-Equivariant Posterior Consistency for Out-of-Distribution Detection in Diffusion Models

  2026cites this paper
• Conditional Denoising Model as a Physical Surrogate Model

  2026cites this paper
• DomusFM: A Foundation Model for Smart-Home Sensor Data

  2026cites this paper
• Realistic image-to-image machine unlearning via decoupling and knowledge retention

  2026cites this paper
• PurSAMERE: Reliable Adversarial Purification via Sharpness-Aware Minimization of Expected Reconstruction Error

  2026cites this paper
• Noise2Score3D: Tweedie's Approach for Unsupervised Point Cloud Denoising

  2025cites this paper
• Unsupervised detection of rare events in liquid biopsy assays

  2025cites this paper
• Model Inversion Attack Against Transfer Learning: Inverting a Model Without Querying It

  2025cites this paper
• FUELVISION: A multimodal data fusion and multimodel ensemble algorithm for wildfire fuels mapping

  2025cites this paper
• Single-Pixel Imaging Based on Enhanced Multi-Network Prior

  2025cites this paper
• Realistic Image-to-Image Machine Unlearning via Decoupling and Knowledge Retention

  2025cites this paper
• Noise2Score3D:Unsupervised Tweedie's Approach for Point Cloud Denoising

  2025cites this paper
• Efficient Deep Equilibrium Models: Denoising Regularization and Average Fixed-Point Initialization to Reduce Function Evaluations

  2025cites this paper
• Classical Autoencoder Distillation of Quantum Adversarial Manipulations

  2025cites this paper
• Privacy-Enhancing Infant Cry Classification with Federated Transformers and Denoising Regularization

  2025cites this paper
• Model-free filtering in high dimensions via projection and score-based diffusions

  2025cites this paper
• Enabling Out-of-Sample Extension in Semi-Supervised Manifold Alignment through Twin Autoencoders

  2025cites this paper
• Quantifying the Ease of Reproducing Training Data in Unconditional Diffusion Models

  2025cites this paper
• RMD-Graph: Adversarial Attacks Resisting Malicious Domain Detection Based on Dual Denoising

  2025cites this paper
• Diagnosing and Improving Diffusion Models by Estimating the Optimal Loss Value

  2025cites this paper
• Denoising Autoencoder for Reconstructing Sensor Observation Data and Predicting Evapotranspiration: Noisy and Missing Values Repair and Uncertainty Quantification

  2025cites this paper
• Stable CDE Autoencoders with Acuity Regularization for Offline Reinforcement Learning in Sepsis Treatment

  2025cites this paper
• Geodesic Calculus on Implicitly Defined Latent Manifolds

  2025cites this paper
• 基于生成式人工智能的计算光学成像进展（特邀）

  2025cites this paper
• FGDCC: Fine-Grained Deep Cluster Categorization - A Framework for Intra-Class Variability Problems in Plant Classification

  2025cites this paper
• Learning What Matters: Steering Diffusion via Spectrally Anisotropic Forward Noise

  2025cites this paper
• A Deep Autoencoder for Fast Spectral-Temporal Fitting of Dynamic Deuterium Metabolic Imaging Data at 7T

  2025cites this paper
• Shaping Inductive Bias in Diffusion Models through Frequency-Based Noise Control

  2025cites this paper
• Score-based Membership Inference on Diffusion Models

  2025influential citation
• Geometric regularity in deterministic sampling dynamics of diffusion-based generative models

  2025cites this paper
• Energy-Tweedie: Score meets Score, Energy meets Energy

  2025cites this paper
• Escaping Plato's Cave: JAM for Aligning Independently Trained Vision and Language Models

  2025cites this paper
• Modeling biodiesel properties by preference learning: Case study of cetane number

  2025cites this paper
• Distributional autoencoders know the score

  2025cites this paper
• Noise & pattern: identity-anchored Tikhonov regularization for robust structural anomaly detection

  2025cites this paper
• InvFusion: Bridging Supervised and Zero-shot Diffusion for Inverse Problems

  2025cites this paper
• An Efficient Echo Reconstruction Method by Physics-Driven Deep Learning

  2025cites this paper
• Navigating the Latent Space Dynamics of Neural Models

  2025influential citation
• Deep learning models for perception of brightness related illusions

  2024cites this paper
• Localized Schrödinger Bridge Sampler

  2024cites this paper
• Deep Network Regularization for Phase-Based Magnetic Resonance Electrical Properties Tomography With Stein's Unbiased Risk Estimator

  2024cites this paper
• PACE: marrying generalization in PArameter-efficient fine-tuning with Consistency rEgularization

  2024cites this paper
• Controllable Unlearning for Image-to-Image Generative Models via ε-Constrained Optimization

  2024cites this paper
• Enhancing Anomaly Detection Generalization through Knowledge Exposure: The Dual Effects of Augmentation

  2024cites this paper
• Image-Based Time Series Forecasting: A Deep Convolutional Neural Network Approach

  2024cites this paper
• Cross-Domain Low-Dose CT Image Denoising With Semantic Preservation and Noise Alignment

  2024cites this paper
• Automatic Eyeblink Artifact Removal from Single Channel EEG Signals Using One-Dimensional Convolutional Denoising Autoencoder

  2024cites this paper
• Deep Few-view High-resolution Photon-counting Extremity CT at Halved Dose for a Clinical Trial

  2024cites this paper
• A Semantic-Aware and Multi-Guided Network for Infrared-Visible Image Fusion

  2024cites this paper
• Bi-Level Motion Imitation for Humanoid Robots

  2024cites this paper
• Enabling Uncertainty Estimation in Iterative Neural Networks

  2024cites this paper
• A qualitative analysis of knowledge graphs in recommendation scenarios through semantics-aware autoencoders

  2024cites this paper
• Masked Graph Autoencoder with Non-discrete Bandwidths

  2024influential citation
• Taming Score-Based Diffusion Priors for Infinite-Dimensional Nonlinear Inverse Problems

  2024cites this paper
• Complete flow characterization from snapshot PIV, fast probes and physics-informed neural networks

  2024cites this paper
• Deep Few-View High-Resolution Photon-Counting CT at Halved Dose for Extremity Imaging

  2024cites this paper
• Self-Supervised Autoencoders for Visual Anomaly Detection

  2024influential citation
• Representation learning with unconditional denoising diffusion models for dynamical systems

  2024cites this paper
• Enhancing Cardiovascular Disease Prediction through Multi-Modal Self-Supervised Learning

  2024cites this paper
• Denoising Variational Graph of Graphs Auto-Encoder for Predicting Structured Entity Interactions

  2024cites this paper
• Client-Customized Adaptation for Parameter-Efficient Federated Learning

  2024cites this paper
• COrAL: Order-Agnostic Language Modeling for Efficient Iterative Refinement

  2024cites this paper
• Nuclear Norm Regularization for Deep Learning

  2024cites this paper
• Towards understanding animal welfare by observing collective flock behaviors via AI-powered Analytics

  2024cites this paper
• Unsupervised Image Denoising with Score Function

  2023cites this paper
• Utility Theory of Synthetic Data Generation

  2023cites this paper
• A Geometric Perspective on Diffusion Models

  2023cites this paper
• Auto-Encoders in Deep Learning—A Review with New Perspectives

  2023cites this paper
• Self-Supervised Learning for Annotation Efficient Biomedical Image Segmentation

  2023cites this paper
• Computer Vision - Statistical Models for Marr's Paradigm

  2023cites this paper
• Neural‐network‐based regularization methods for inverse problems in imaging

  2023cites this paper
• Universal Smoothed Score Functions for Generative Modeling

  2023cites this paper
• Towards Predicting Equilibrium Distributions for Molecular Systems with Deep Learning

  2023cites this paper
• Human internal state estimation as blind source separation using a dynamic auto-encoder

  2023influential citation
• Interpreting denoising autoencoders with complex perturbation approach

  2023cites this paper
• Foundations of machine learning for low-temperature plasmas: methods and case studies

  2023cites this paper
• Independent and Collaborative Demosaicking Neural Networks

  2023cites this paper
• Bayes-Optimal Unsupervised Learning for Channel Estimation in Near-Field Holographic MIMO

  2023cites this paper
• Self-Supervised Deep Learning for Image Reconstruction: A Langevin Monte Carlo Approach

  2023cites this paper
• Additive autoencoder for dimension estimation

  2023cites this paper
• Time and temporal abstraction in continual learning: tradeoffs, analogies and regret in an active measuring setting

  2023cites this paper
• Learning binary codes for fast image retrieval with sparse discriminant analysis and deep autoencoders

  2023cites this paper
• Echoes in the Noise: Posterior Samples of Faint Galaxy Surface Brightness Profiles with Score-based Likelihoods and Priors

  2023cites this paper
• Learning Bayes-Optimal Channel Estimation for Holographic MIMO in Unknown EM Environments

  2023cites this paper
• Provable Probabilistic Imaging Using Score-Based Generative Priors

  2023cites this paper
• Analyzing Multimodal Probability Measures with Autoencoders.

  2023cites this paper
• How do Minimum-Norm Shallow Denoisers Look in Function Space?

  2023cites this paper
• Compressive Reconstruction Based on Sparse Autoencoder Network Prior for Single-Pixel Imaging

  2023cites this paper
• Bayesian Imaging for Radio Interferometry with Score-Based Priors

  2023cites this paper
• Estimation and analysis of insect population dynamics parameters via physiologically based models and hybrid genetic algorithm MCMC methods

  2023cites this paper
• Diffusion Model as Representation Learner

  2023cites this paper
• Neural Image Compression: Generalization, Robustness, and Spectral Biases

  2023cites this paper
• A service composition evolution method that combines deep clustering and a service requirement context model

  2023cites this paper
• Fast Diffusion-Based Counterfactuals for Shortcut Removal and Generation

  2023cites this paper
• A Comprehensive Review of Emerging Trends in Aircraft Structural Prognostics and Health Management

  2023cites this paper
• Targeted Collapse Regularized Autoencoder for Anomaly Detection: Black Hole at the Center

  2023cites this paper
• Protein Discovery with Discrete Walk-Jump Sampling

  2023cites this paper
• L EARNING PROTEIN FAMILY MANIFOLDS WITH SMOOTHED ENERGY - BASED MODELS

  2023cites this paper
• On Explicit Curvature Regularization in Deep Generative Models

  2023cites this paper
• Geometrically regularized autoencoders for non-Euclidean data

  2023cites this paper