MADE: Masked Autoencoder for Distribution Estimation

There has been a lot of recent interest in designing neural network models to estimate a distribution from a set of examples. We introduce a simple modification for autoencoder neural networks that yields powerful generative models. Our method masks the autoencoder's parameters to respect autoregressive constraints: each input is reconstructed only from previous inputs in a given ordering. Constrained this way, the autoencoder outputs can be interpreted as a set of conditional probabilities, and their product, the full joint probability. We can also train a single network that can decompose the joint probability in multiple different orderings. Our simple framework can be applied to multiple architectures, including deep ones. Vectorized implementations, such as on GPUs, are simple and fast. Experiments demonstrate that this approach is competitive with state-of-the-art tractable distribution estimators. At test time, the method is significantly faster and scales better than other autoregressive estimators.

• Publication year

  2015

• Venue

  International Conference on Machine Learning

• Publication date

  2015-02-11

• Fields of study

  Mathematics, Computer Science

• Identifiers
  arXiv 1502.03509
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Modelling acoustic feature dependencies with artificial neural networks: Trajectory-RNADE

  2015cited by this paper
• Dropout: a simple way to prevent neural networks from overfitting

  2014cited by this paper
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models

  2014cited by this paper
• NICE: Non-linear Independent Components Estimation

  2014influential reference
• Auto-Encoding Variational Bayes

  2013cited by this paper
• Regularization of Neural Networks using DropConnect

  2013cited by this paper
• Deep Generative Stochastic Networks Trainable by Backprop

  2013cited by this paper
• A Deep and Tractable Density Estimator

  2013influential reference
• RNADE: The real-valued neural autoregressive density-estimator

  2013cited by this paper
• Deep AutoRegressive Networks

  2013cited by this paper
• Theano: new features and speed improvements

  2012cited by this paper
• ADADELTA: An Adaptive Learning Rate Method

  2012influential reference
• Learning Representations by Maximizing Compression

  2011cited by this paper
• Adaptive Subgradient Methods for Online Learning and Stochastic Optimization

  2011cited by this paper
• Sum-product networks: A new deep architecture

  2011cited by this paper
• The Neural Autoregressive Distribution Estimator

  2011cited by this paper
• On the quantitative analysis of deep belief networks

  2008cited by this paper
• Generative versus discriminative training of RBMs for classification of fMRI images

  2008cited by this paper
• Modeling High-Dimensional Discrete Data with Multi-Layer Neural Networks

  1999influential reference

• Selecting Optimal Variable Order in Autoregressive Ising Models

  2026influential citation
• Mutual Calibration Network for Multi-View Clustering

  2026cites this paper
• A Dual-Layer End-to-End Cost Estimation Model for LSM-Tree-Based Database Systems

  2026cites this paper
• ScDiVa: Masked Discrete Diffusion for Joint Modeling of Single-Cell Identity and Expression

  2026cites this paper
• Deep probabilistic circuits for efficient uncertainty handling in learning systems

  2026cites this paper
• Detecting AI-generated essays using fine-tuned XLNet-CNN hybrid techniques: a study of the academic integrity challenge

  2026cites this paper
• Learning at the Edge: Tailed-Uniform Sampling for Robust Simulation-Based Inference

  2026cites this paper
• Scalable Uncertainty Quantification for Black-Box Density-Based Clustering

  2026cites this paper
• Normalizing flows for Bayesian parameter inference in computational mechanics

  2026cites this paper
• Leveraging recurrence in neural network wavefunctions for large-scale simulations of Heisenberg antiferromagnets on the square lattice

  2025cites this paper
• A Dual-Filtering Approach for Malicious Traffic Detection Against Label Noise and Class Imbalance Issues

  2025cites this paper
• Inverse Autoregressive Flows for Zero Degree Calorimeter fast simulation

  2025cites this paper
• CUBE: A Cardinality Estimator Based on Neural CDF

  2025cites this paper
• A Cardinality Estimation Model Based on Orthogonal Matching Pursuit and Sparse Attention Mechanism

  2025cites this paper
• Neural Autoregressive Flows for Markov Boundary Learning

  2025influential citation
• Large-Scale Neural Network Quantum States Calculation for Quantum Chemistry on a New Sunway Supercomputer

  2025cites this paper
• Demonstrating Real Advantage of Machine-Learning-Enhanced Monte Carlo for Combinatorial Optimization

  2025cites this paper
• Toward Effective Knowledge Distillation: Navigating Beyond Small-Data Pitfall

  2025cites this paper
• Sequential keypoint density estimator: an overlooked baseline of skeleton-based video anomaly detection

  2025influential citation
• Large Language Model Scaling Laws for Neural Quantum States in Quantum Chemistry

  2025cites this paper
• Challenges and Solutions in Selecting Optimal Lossless Data Compression Algorithms

  2025cites this paper
• Structure-Fair Quantum Circuit Complexity: An Auditable Information-Theoretic Lower Bound

  2025cites this paper
• Learning optimal summary statistics of galaxy catalogs with SBI

  2025cites this paper
• Variational Autoregressive Networks Applied to $\phi^4$ Field Theory Systems

  2025cites this paper
• An Overview of Deep Generative Models-Introduction, Applications and Recent Advances

  2025cites this paper
• IrisML: Neural Posterior Estimation for the Spectral Energy Distribution fitting

  2025cites this paper
• SECRET: Stochasticity Emulator for Cosmic Ray Electrons

  2025influential citation
• Anomaly detection in broadband networks: Using normalizing flows for multivariate time series

  2025cites this paper
• CausalMan: A physics-based simulator for large-scale causality

  2025influential citation
• CroTad: A Contrastive Reinforcement Learning Framework for Online Trajectory Anomaly Detection

  2025cites this paper
• Variational Transdimensional Inference

  2025influential citation
• Reviving Any-Subset Autoregressive Models with Principled Parallel Sampling and Speculative Decoding

  2025cites this paper
• Hybrid Bernstein Normalizing Flows for Flexible Multivariate Density Regression with Interpretable Marginals

  2025influential citation
• Quirk SUEP

  2025cites this paper
• Gaussian Process Tilted Nonparametric Density Estimation Using Fisher Divergence Score Matching

  2025cites this paper
• Accelerate TarFlow Sampling with GS-Jacobi Iteration

  2025cites this paper
• Performance of machine-learning-assisted Monte Carlo in sampling from simple statistical physics models.

  2025cites this paper
• Estimation of the reduced density matrix and entanglement entropies using autoregressive networks

  2025cites this paper
• Foundation Artificial Intelligence Models for Health Recognition Using Face Photographs (FAHR-Face)

  2025cites this paper
• Flow Stochastic Segmentation Networks

  2025cites this paper
• Multi-Granularity Distribution Modeling for Video Watch Time Prediction via Exponential-Gaussian Mixture Network

  2025cites this paper
• Neural posterior unfolding

  2025cites this paper
• Structure-Fair Quantum Circuit Complexity: An Information-Theoretic Lower Bound

  2025cites this paper
• Improving adversarial transferability via semantic-style joint expectation perturbations

  2025cites this paper
• Efficient Autoregressive Inference for Transformer Probabilistic Models

  2025cites this paper
• Benchmarking 3D generative autoencoders for pseudo-healthy reconstruction of brain 18F-fluorodeoxyglucose positron emission tomography

  2025cites this paper
• DDN-Turbo: Enhanced Deconvolutional Density Networks for Data-Scarce Conditional Density Estimation

  2025cites this paper
• Tracking large chemical reaction networks and rare events by neural networks

  2025cites this paper
• Graph Autoencoder and StrNN based Causal Analysis of Mortality in Heart Failure Patients

  2025cites this paper
• CoLSE: A Lightweight and Robust Hybrid Learned Model for Single-Table Cardinality Estimation using Joint CDF

  2025cites this paper
• Generative Decoding for Quantum Error-correcting Codes

  2025influential citation
• Multi-Flow: Multi-View-Enriched Normalizing Flows for Industrial Anomaly Detection

  2025cites this paper
• Machine learning stochastic dynamics

  2025cites this paper
• Attitude Detection Model Based on Improved High Resolution Network

  2025cites this paper
• Hierarchical autoregressive neural networks in three-dimensional statistical system

  2025cites this paper
• Exploring the BSM parameter space with neural network aided Simulation-Based Inference

  2025cites this paper
• DSDIR: A Two-Stage Method for Addressing Noisy Long-Tailed Problems in Malicious Traffic Detection

  2025cites this paper
• A Real-time Anomaly Detection Method for Robots based on a Flexible and Sparse Latent Space

  2025influential citation
• Deep inference of simulated strong lenses in ground-based surveys

  2025cites this paper
• Fast Simulation of the Zero Degree Calorimeter Responses with Generative Neural Networks

  2025cites this paper
• FedWeight: mitigating covariate shift of federated learning on electronic health records data through patients re-weighting

  2025cites this paper
• SSIM: self-supervised learning method based on spatially selected shifts and irregular image masking

  2025cites this paper
• Sketched Sum-Product Networks for Joins

  2025cites this paper
• Probabilistic memory auto-encoding network for abnormal behavior detection in surveillance video

  2025cites this paper
• DistJoin: A Decoupled Join Cardinality Estimator Based on Adaptive Neural Predicate Modulation

  2025cites this paper
• Multilevel Generative Samplers for Investigating Critical Phenomena

  2025cites this paper
• EBM-WGF: Training energy-based models with Wasserstein gradient flow

  2025cites this paper
• DeCaFlow: A Deconfounding Causal Generative Model

  2025cites this paper
• Paraflow: fast calorimeter simulations parameterized in upstream material configurations

  2025cites this paper
• Integrating neural networks and tensor networks for computing free energy

  2025cites this paper
• Conditional attention guided normalizing flow for low-light image enhancement

  2025cites this paper
• Dimensionality reduction in 3D causal deep learning for neuroimage generation: an evaluation study

  2025cites this paper
• A Unified Model for Cardinality Estimation by Learning from Data and Queries via Sum-Product Networks

  2025cites this paper
• Spatial–Temporal Heatmap Masked Autoencoder for Skeleton-Based Action Recognition

  2025cites this paper
• Multivariate Latent Recalibration for Conditional Normalizing Flows

  2025cites this paper
• Causal Posterior Estimation

  2025cites this paper
• BIMA: Bijective Maximum Likelihood Learning Approach to Hallucination Prediction and Mitigation in Large Vision-Language Models

  2025cites this paper
• Model Informed Flows for Bayesian Inference of Probabilistic Programs

  2025cites this paper
• GMM-cGAN: Mitigating data scarcity and label noise for robust encrypted malicious traffic classification

  2025cites this paper
• Data-Agnostic Cardinality Learning from Imperfect Workloads

  2025cites this paper
• Simulation-Free Differential Dynamics through Neural Conservation Laws

  2025cites this paper
• Any-Order GPT as Masked Diffusion Model: Decoupling Formulation and Architecture

  2025cites this paper
• EXPO: Stable Reinforcement Learning with Expressive Policies

  2025cites this paper
• Observable optimization for precision theory: machine learning energy correlators

  2025cites this paper
• Bayesian Insights into Exchange and Restriction in Gray Matter Diffusion MRI

  2025cites this paper
• A Semantic Inpainting Framework for Distributed Cross-Modal Integrated Sensing and Communication

  2025cites this paper
• Modeling nonstationary spatial processes with normalizing flows

  2025cites this paper
• A Quasi-Monte Carlo Method Based on Neural Autoregressive Flow

  2025cites this paper
• BatchTNMC: Efficient sampling of two-dimensional spin glasses using tensor network Monte Carlo

  2025cites this paper
• Physics-constrained normalizing flow for identification and modeling of vortex-induced vibration in stay cables

  2025cites this paper
• Black-Box Combinatorial Optimization with Order-Invariant Reinforcement Learning

  2025cites this paper
• Universality and kernel-adaptive training for classically trained, quantum-deployed generative models

  2025cites this paper
• $\texttt{SBi3PCF:}$ Simulation-based inference with the integrated 3PCF

  2025cites this paper
• Meta-Learning Universal Priors Using Non-Injective Change of Variables

  2024cites this paper
• Faster and More Efficient Subject Image Generation for Text-to-Image Diffusion Models

  2024cites this paper
• Variational Evolutionary Network for Statistical Physics Systems

  2024cites this paper
• CauImputation: Utilizing Structural Causal Model in Data Imputation

  2024cites this paper
• Deep STI: Deep Stochastic Time-series Imputation on Electronic Health Records

  2024cites this paper
• Normalizing Flows are Capable Generative Models

  2024cites this paper
• Empirical evaluation of normalizing flows in Markov chain Monte Carlo

  2024cites this paper