Representation Learning: A Review and New Perspectives

The success of machine learning algorithms generally depends on data representation, and we hypothesize that this is because different representations can entangle and hide more or less the different explanatory factors of variation behind the data. Although specific domain knowledge can be used to help design representations, learning with generic priors can also be used, and the quest for AI is motivating the design of more powerful representation-learning algorithms implementing such priors. This paper reviews recent work in the area of unsupervised feature learning and deep learning, covering advances in probabilistic models, autoencoders, manifold learning, and deep networks. This motivates longer term unanswered questions about the appropriate objectives for learning good representations, for computing representations (i.e., inference), and the geometrical connections between representation learning, density estimation, and manifold learning.

• Publication year

  2012

• Venue

  IEEE Transactions on Pattern Analysis and Machine Intelligence

• Publication date

  2012-06-24

• Fields of study

  Mathematics, Computer Science, Medicine

• Identifiers
  DOI 10.1109/TPAMI.2013.50arXiv 1206.5538PMID 23787338
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar, PubMed

• No claims are published for this paper.

• No concepts are published for this paper.

• Mathematik in den Naturwissenschaften Leipzig When Does a Mixture of Products Contain a Product of Mixtures ?

  2014cited by this paper
• Training Recurrent Neural Networks

  2013cited by this paper
• Natural Gradient Revisited

  2013cited by this paper
• Revisiting Natural Gradient for Deep Networks

  2013cited by this paper
• Structured Output Layer Neural Network Language Models for Speech Recognition

  2013cited by this paper
• Implicit Density Estimation by Local Moment Matching to Sample from Auto-Encoders

  2012influential reference
• Spike-and-Slab Sparse Coding for Unsupervised Feature Discovery

  2012influential reference
• Practical Bayesian Optimization of Machine Learning Algorithms

  2012cited by this paper
• Large, Pruned or Continuous Space Language Models on a GPU for Statistical Machine Translation

  2012cited by this paper
• Conversational Speech Transcription Using Context-Dependent Deep Neural Networks

  2012influential reference
• ImageNet classification with deep convolutional neural networks

  2012influential reference
• Multiple Texture Boltzmann Machines

  2012influential reference
• Deep Learning Made Easier by Linear Transformations in Perceptrons

  2012cited by this paper
• Acoustic Modeling Using Deep Belief Networks

  2012cited by this paper
• Multi-column deep neural networks for image classification

  2012cited by this paper
• Context-Dependent Pre-Trained Deep Neural Networks for Large-Vocabulary Speech Recognition

  2012cited by this paper
• Deep Neural Networks for Acoustic Modeling in Speech Recognition

  2012cited by this paper
• On Training Deep Boltzmann Machines

  2012cited by this paper
• Better Mixing via Deep Representations

  2012influential reference
• A Generative Process for Contractive Auto-Encoders

  2012influential reference
• An Efficient Learning Procedure for Deep Boltzmann Machines

  2012cited by this paper
• Random Search for Hyper-Parameter Optimization

  2012cited by this paper
• Efficient BackProp

  2012influential reference
• Practical Recommendations for Gradient-Based Training of Deep Architectures

  2012influential reference
• What regularized auto-encoders learn from the data-generating distribution

  2012influential reference
• Multimodal learning with deep Boltzmann machines

  2012cited by this paper
• How does the brain solve visual object recognition?

  2012cited by this paper
• Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation and Transcription

  2012influential reference
• Réseaux de neurones à relaxation entraînés par critère d'autoencodeur débruitant

  2012influential reference
• A Practical Guide to Training Restricted Boltzmann Machines

  2012influential reference
• Neural Networks: Tricks of the Trade

  2012cited by this paper
• Marginalized Denoising Autoencoders for Domain Adaptation

  2012influential reference
• Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups

  2012cited by this paper
• Joint Learning of Words and Meaning Representations for Open-Text Semantic Parsing

  2012cited by this paper
• Structured sparsity through convex optimization

  2011influential reference
• Learning hierarchical invariant spatio-temporal features for action recognition with independent subspace analysis

  2011influential reference
• Should Penalized Least Squares Regression be Interpreted as Maximum A Posteriori Estimation?

  2011cited by this paper
• Natural Language Processing (Almost) from Scratch

  2011cited by this paper
• Deep Sparse Rectifier Neural Networks

  2011influential reference
• Transforming Auto-Encoders

  2011influential reference
• Higher Order Contractive Auto-Encoder

  2011influential reference
• Enhanced Gradient and Adaptive Learning Rate for Training Restricted Boltzmann Machines

  2011cited by this paper
• The Importance of Encoding Versus Training with Sparse Coding and Vector Quantization

  2011influential reference
• Feature engineering in Context-Dependent Deep Neural Networks for conversational speech transcription

  2011influential reference
• On Tracking The Partition Function

  2011cited by this paper
• Probabilistic Inference Using Markov Chain Monte Carlo Methods

  2011influential reference
• Closed-Form EM for Sparse Coding and Its Application to Source Separation

  2011cited by this paper
• Quickly Generating Representative Samples from an RBM-Derived Process

  2011influential reference
• Contractive Auto-Encoders: Explicit Invariance During Feature Extraction

  2011influential reference
• Evaluating Bayesian and L1 Approaches for Sparse Unsupervised Learning

  2011cited by this paper
• From machine learning to machine reasoning

  2011cited by this paper
• Large-Scale FPGA-based Convolutional Networks

  2011influential reference
• Empirical Evaluation and Combination of Advanced Language Modeling Techniques

  2011cited by this paper
• A Spike and Slab Restricted Boltzmann Machine

  2011influential reference
• Learning Deep Energy Models

  2011influential reference
• On deep generative models with applications to recognition

  2011cited by this paper
• Ask the locals: Multi-way local pooling for image recognition

  2011cited by this paper
• Deep Learning of Representations for Unsupervised and Transfer Learning

  2011influential reference
• Selecting Receptive Fields in Deep Networks

  2011influential reference
• On optimization methods for deep learning

  2011influential reference
• Spike and Slab Variational Inference for Multi-Task and Multiple Kernel Learning

  2011cited by this paper
• Structured sparse coding via lateral inhibition

  2011influential reference
• Temporal Pooling and Multiscale Learning for Automatic Annotation and Ranking of Music Audio

  2011cited by this paper
• Group Invariant Scattering

  2011cited by this paper
• Unsupervised Models of Images by Spikeand-Slab RBMs

  2011influential reference
• On the Expressive Power of Deep Architectures

  2011cited by this paper
• A Connection Between Score Matching and Denoising Autoencoders

  2011influential reference
• On Autoencoders and Score Matching for Energy Based Models

  2011cited by this paper
• Domain Adaptation for Large-Scale Sentiment Classification: A Deep Learning Approach

  2011cited by this paper
• Large-Scale Learning of Embeddings with Reconstruction Sampling

  2011influential reference
• Unsupervised learning of visual invariance with temporal coherence

  2011cited by this paper
• On the training of recurrent neural networks

  2011influential reference
• Learning image representations from the pixel level via hierarchical sparse coding

  2011cited by this paper
• Dynamic Pooling and Unfolding Recursive Autoencoders for Paraphrase Detection

  2011cited by this paper
• Algorithms for Hyper-Parameter Optimization

  2011cited by this paper
• Semi-Supervised Recursive Autoencoders for Predicting Sentiment Distributions

  2011cited by this paper
• Unsupervised Learning of Sparse Features for Scalable Audio Classification

  2011cited by this paper
• A Closed-Form EM Algorithm for Sparse Coding

  2011cited by this paper
• ICA with Reconstruction Cost for Efficient Overcomplete Feature Learning

  2011cited by this paper
• Learning Recurrent Neural Networks with Hessian-Free Optimization

  2011cited by this paper
• The Manifold Tangent Classifier

  2011influential reference
• Empirical Risk Minimization of Graphical Model Parameters Given Approximate Inference, Decoding, and Model Structure

  2011cited by this paper
• Unsupervised and Transfer Learning Challenge: a Deep Learning Approach

  2011influential reference
• Asymptotic Efficiency of Deterministic Estimators for Discrete Energy-Based Models: Ratio Matching and Pseudolikelihood

  2011cited by this paper
• Improved Local Coordinate Coding using Local Tangents

  2010cited by this paper
• Deep Coding Network

  2010cited by this paper
• Deep learning via Hessian-free optimization

  2010influential reference
• Deconvolutional networks

  2010cited by this paper
• DECISION TREES DO NOT GENERALIZE TO NEW VARIATIONS

  2010cited by this paper
• Large scale image annotation: learning to rank with joint word-image embeddings

  2010cited by this paper
• A Theoretical Analysis of Feature Pooling in Visual Recognition

  2010cited by this paper
• Phone Recognition with the Mean-Covariance Restricted Boltzmann Machine

  2010cited by this paper
• Factored 3-Way Restricted Boltzmann Machines For Modeling Natural Images

  2010influential reference
• Learning Fast Approximations of Sparse Coding

  2010cited by this paper
• Why Does Unsupervised Pre-training Help Deep Learning?

  2010influential reference
• Sample Complexity of Testing the Manifold Hypothesis

  2010cited by this paper
• Deep, Big, Simple Neural Nets for Handwritten Digit Recognition

  2010influential reference
• Classification with scattering operators

  2010cited by this paper
• Emergence of Complex-Like Cells in a Temporal Product Network with Local Receptive Fields

  2010cited by this paper
• Generating more realistic images using gated MRF's

  2010influential reference

• KGC-Explainer: Toward Explainable Knowledge Graph Completion

  2026cites this paper
• OSNIP: Breaking the Privacy-Utility-Efficiency Trilemma in LLM Inference via Obfuscated Semantic Null Space

  2026cites this paper
• Assessing mindset states of Hong Kong secondary students using machine learning in real-world online learning environment

  2026cites this paper
• Enabling AI-driven modular building design: an auto-decoder approach for IFC 3D geometry representation

  2026cites this paper
• Rethinking Intelligence: Brain-like Neuron Network

  2026cites this paper
• LATS: Robust Trajectory Similarity Computation via Hybrid LSTM-Attention and Adaptive Contrastive Learning

  2026cites this paper
• DUALF-D: Disentangled dual-hyperprior approach for light field image compression

  2026cites this paper
• Deep Reinforcement Learning-Based DFE Structure Optimization for High-Speed DRAM Signals

  2026cites this paper
• Multi-stage representation learning for blind Room-Acoustic parameter estimation with uncertainty quantification.

  2026cites this paper
• Towards Trustworthy Urban Land Use Classification: A Synergistic Fusion of Deep Learning and Explainable Machine Learning with a Nanning Case Study

  2026cites this paper
• Zero-Shot Deception Jamming Recognition Based on Multipulse Integrated Spectrum Reconstruction

  2026cites this paper
• The ghost of behaviorism: critical reflections on methodological limitations in the research of large language models psychology

  2026cites this paper
• Bridging Functional and Representational Similarity via Usable Information

  2026cites this paper
• Neighborhood α -weakly revealing conditions and distinguishing neighborhood-infer exploration methods in reinforcement learning

  2026cites this paper
• CausalSymptom: Learning Causal Disentangled Representation for Depression Severity Estimation on Transcribed Clinical Interviews

  2026cites this paper
• AI-Driven Multiagent IoT System for Energy Consumption Anomaly Detection

  2026cites this paper
• A predictive coding model for online sentence processing

  2026cites this paper
• Multifeature Fusion Radar Detection for Maritime Small Targets via Disentangled Representation Learning

  2026cites this paper
• Enhancing Alzheimer’s disease classification with a transformer-based model using self-supervised learning

  2026cites this paper
• Leveraging a Repetitive Filtering Framework for Robust Pulse Stream Extraction in Distributed Optical Fiber Sensing Signals

  2026cites this paper
• Universal Latent Homeomorphic Manifolds: A Framework for Cross-Domain Representation Unification

  2026cites this paper
• Transforming Drilling Textbooks into Semantic Tools for Design Verification and Knowledge Extraction

  2026cites this paper
• Actionable Interpretability Must Be Defined in Terms of Symmetries

  2026cites this paper
• Clinically-guided models or foundation models? predicting cervical spondylotic myelopathy from electronic health records

  2026cites this paper
• Energy consumption pattern recognition and reduction in milling operations via deep neural networks

  2026cites this paper
• GO-OSC and VASH: Geometry-Aware Representation Learning for Early Degradation Detection in Oscillatory Systems

  2026cites this paper
• A Machine Learning Pipeline for Cusp Height Prediction in Worn Lower Molars: Methodological Proof-of-Concept and Validation Across Homo

  2026cites this paper
• XFACTORS: Disentangled Information Bottleneck via Contrastive Supervision

  2026cites this paper
• Representation Unlearning: Forgetting through Information Compression

  2026cites this paper
• A Bayesian Backed DeepLabV3+ Customized Hybrid-Depth Framework for Brain Tumor Segmentation and Classification

  2026cites this paper
• Advanced Learning Algorithms for Integrated Sensing and Communication (ISAC) Systems in 6G and Beyond: A Comprehensive Survey

  2026cites this paper
• Data-driven fault detection framework for wheel speed sensor in Heavy Road Vehicles

  2026cites this paper
• MAIR: Model Agnostic Instance Reweighing for Heterogeneous Federated Learning

  2026cites this paper
• Federated learning-based regularized adversarial graph embedding for cross-social network user alignment

  2026cites this paper
• Learning diverse and adaptive representations for continual learning

  2026cites this paper
• Assessing airflow perturbations in naturally ventilated barns: A comparative study of image similarity metrics using machine learning

  2026cites this paper
• CDRNet: A Causality Disentanglement Few-Shot Mechanical Fault Diagnosis

  2026cites this paper
• Bridging data gaps for EV demand estimation: A transfer learning approach

  2026cites this paper
• ECR: Manifold-Guided Semantic Cues for Compact Language Models

  2026cites this paper
• HEEGNet: Hyperbolic Embeddings for EEG

  2026cites this paper
• Learnable protein representations in computational biology for predicting drug-target affinity

  2026cites this paper
• Average Volatility Dimensioning (AVD): dimension reduction technique for multivariate time series

  2026cites this paper
• Unsupervised Variational-Autoencoder-Based Analysis of Morphological Representations in Magnetic-Nanoparticle-Treated Macrophages

  2026cites this paper
• Residual Power Flow for Neural Solvers

  2026cites this paper
• Enhancing Visual In-Context Learning by Multi-Faceted Fusion

  2026cites this paper
• Condition-aware AI framework for automated structural health monitoring

  2026cites this paper
• Information Theoretic Perspective on Representation Learning

  2026cites this paper
• Semidefinite Programming for Quantum Channel Learning

  2026cites this paper
• Inverting Self-Organizing Maps: A Unified Activation-Based Framework

  2026cites this paper
• Training instability in deep learning follows low-dimensional dynamical principles

  2026cites this paper
• The impact of AI on modern oncology from early detection to personalized cancer treatment

  2026cites this paper
• A graph neural network model for wave buoy data imputation using spatiotemporal information

  2026cites this paper
• Latent-Space Contrastive Reinforcement Learning for Stable and Efficient LLM Reasoning

  2026cites this paper
• What Do Learned Models Measure?

  2026cites this paper
• Machine Learning‐Based Data Deduplication: Techniques, Challenges, and Future Directions

  2026cites this paper
• An explainable framework for the relationship between dementia and metabolism patterns

  2026cites this paper
• Point Cloud Registration and Necking Defect Detection for Sheet Metal Quality Inspection

  2026cites this paper
• Tabular and graph-based representations for noise and missing data in robust machine learning

  2026cites this paper
• FlexCausal: Flexible Causal Disentanglement via Structural Flow Priors and Manifold-Aware Interventions

  2026cites this paper
• The Patient is not a Moving Document: A World Model Training Paradigm for Longitudinal EHR

  2026cites this paper
• Low-Rank Plus Sparse Matrix Transfer Learning under Growing Representations and Ambient Dimensions

  2026cites this paper
• Multimodal emotion recognition with high-level feature fusion of audio and text via cross-attention

  2026cites this paper
• A Decade of Video Analytics at Edge: Training, Deployment, Orchestration, and Platforms

  2026cites this paper
• New idtracker.ai: rethinking multi-animal tracking as a representation learning problem to increase accuracy and reduce tracking times

  2026cites this paper
• Insights into the impact of visual and textual information on investment decision-making: A multimodal business plan analysis via deep representation learning

  2026cites this paper
• Bandwidth Management in Semantic Communications: A Tradeoff Between Data Sensing and Transmission

  2026cites this paper
• Enhancing neural topic modeling for social media text via semantic bag of word clusters and log-domain Sinkhorn transport

  2026cites this paper
• Multimodal fusion in speech emotion recognition: A comprehensive review of methods and technologies

  2026cites this paper
• Self-Supervised Disentangled Representation Learning via Compositional Invariance

  2026cites this paper
• A two-stage detection method for vessel targets in sea clutter based on time-frequency feature anomalies in shipborne HFSWR

  2026cites this paper
• Mapping lithium hosting claystone using WorldView-3 image and deep learning method

  2026cites this paper
• A synergistic approach to decoupling and knowledge distillation for long- and short-term knowledge states in knowledge tracing

  2026cites this paper
• Tillage-induced soil feature extraction and multi-sensors fusion for tillage system classification

  2026cites this paper
• Gaussian enhanced deep reinforcement learning for USV navigation with scarce environmental priors

  2026cites this paper
• IGrNet: A robust graph neural network framework for classifying NURBS-based elements in isogeometric analysis with application to contact mechanics

  2026cites this paper
• Machine Learning-based Reduced-Order Model and multi-fidelity approaches for field reconstruction in CFD: A review

  2026cites this paper
• Why motor learning involves multiple systems: an algorithmic perspective

  2026cites this paper
• WCK-SSAM: Working condition knowledge guided sparse self-attention mechanism for operational status perception of multi-modal fused magnesium furnace

  2026cites this paper
• GAN-Guided Unsupervised, Wrapper Feature Selector for PolSAR Image

  2026cites this paper
• Manhole Localization and Condition Diagnostics in Telecom Networks Using Distributed Acoustic and Temperature Sensing

  2026cites this paper
• Structural Representations for Cross-Attack Generalization in AI Agent Threat Detection

  2026cites this paper
• Accelerated Full Waveform Inversion by Deep Compressed Learning

  2026cites this paper
• From Parametric Determinism to Emergent Fusion: Data-Curated Style Control in Connectionist Architecture

  2026cites this paper
• Intelligence as high-dimensional coherence: The observable dimensionality bound and computational tractability

  2026cites this paper
• Learning Reconstructive Embeddings in Reproducing Kernel Hilbert Spaces via the Representer Theorem

  2026cites this paper
• A Causal Information-Flow Framework for Unbiased Learning-to-Rank

  2026cites this paper
• Applying Embedding-Based Retrieval to Airbnb Search

  2026cites this paper
• CoRe: Learning compact representation with prior-guided initialization

  2026cites this paper
• Task-free continual learning with expansion-based granular CNN: Gradual partitioning of manifolds in image stream classification

  2026cites this paper
• A Highly Efficient Diversity-based Input Selection for DNN Improvement Using VLMs

  2026cites this paper
• Learning Domain-Invariant Representations for Cross-Domain Image Registration via Scene-Appearance Disentanglement

  2026cites this paper
• Adaptive Label Error Detection: A Bayesian Approach to Mislabeled Data Detection

  2026cites this paper
• Searching for Quantum Effects in the Brain: A Bell-Type Test for Nonclassical Latent Representations in Autoencoders

  2026cites this paper
• Learning object representations through amortized inference over probabilistic programs

  2026cites this paper
• Parallel Clusters: Visual Comparison of Embeddings Based on Multi-Scale Neighborhood Analysis

  2026cites this paper
• Ensemble Transitive Bidirectional Decoupled Self-Distillation for Time-Series Classification

  2026cites this paper
• A comprehensive survey on artificial intelligence for biomolecule design

  2026cites this paper
• Newton downhill optimizer with application to engineering optimization and breast cancer feature selection

  2026cites this paper
• The Geometry of Thought: How Scale Restructures Reasoning In Large Language Models

  2026cites this paper
• Cross-domain neural alignment (CDNA): A deep supervised domain adaptation in human activity recognition using device-free sensing

  2026cites this paper