Learning with hidden variables

Learning and inferring features that generate sensory input is a task continuously performed by cortex. In recent years, novel algorithms and learning rules have been proposed that allow neural network models to learn such features from natural images, written text, audio signals, etc. These networks usually involve deep architectures with many layers of hidden neurons. Here we review recent advancements in this area emphasizing, amongst other things, the processing of dynamical inputs by networks with hidden nodes and the role of single neuron models. These points and the questions they arise can provide conceptual advancements in understanding of learning in the cortex and the relationship between machine learning approaches to learning with hidden nodes and those in cortical circuits.

• Publication year

  2015

• Venue

  Current Opinion in Neurobiology

• Publication date

  2015-06-01

• Fields of study

  Biology, Physics, Computer Science, Mathematics, Medicine

• Identifiers
  DOI 10.1016/j.conb.2015.07.006arXiv 1506.00354PMID 26298193
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar, PubMed

• No claims are published for this paper.

• No concepts are published for this paper.

• Towards Biologically Plausible Deep Learning

  2015cited by this paper
• Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

  2015cited by this paper
• Spiking neuron network Helmholtz machine

  2015influential reference
• A Simple Way to Initialize Recurrent Networks of Rectified Linear Units

  2015cited by this paper
• Target Propagation

  2015cited by this paper
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention

  2015cited by this paper
• Difference Target Propagation

  2014cited by this paper
• Show and tell: A neural image caption generator

  2014cited by this paper
• Large-Scale Video Classification with Convolutional Neural Networks

  2014cited by this paper
• Neural Variational Inference and Learning in Belief Networks

  2014cited by this paper
• RECURRENT NEURAL NETWORKS

  2014cited by this paper
• Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)

  2014cited by this paper
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models

  2014cited by this paper
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling

  2014cited by this paper
• Two-Stream Convolutional Networks for Action Recognition in Videos

  2014cited by this paper
• Deep Speech: Scaling up end-to-end speech recognition

  2014cited by this paper
• Identifying and attacking the saddle point problem in high-dimensional non-convex optimization

  2014cited by this paper
• CNN Features Off-the-Shelf: An Astounding Baseline for Recognition

  2014cited by this paper
• Belief propagation and replicas for inference and learning in a kinetic Ising model with hidden spins

  2014cited by this paper
• Sequence to Sequence Learning with Neural Networks

  2014cited by this paper
• Inferring hidden states in a random kinetic Ising model: replica analysis

  2014cited by this paper
• Multi-Prediction Deep Boltzmann Machines

  2013cited by this paper
• Learning Hierarchical Features for Scene Labeling

  2013cited by this paper
• Speech recognition with deep recurrent neural networks

  2013cited by this paper
• Generating Sequences With Recurrent Neural Networks

  2013cited by this paper
• Auto-Encoding Variational Bayes

  2013cited by this paper
• Learning with Hierarchical-Deep Models

  2013cited by this paper
• Deep Generative Stochastic Networks Trainable by Backprop

  2013cited by this paper
• Learning and inference in a nonequilibrium Ising model with hidden nodes.

  2013cited by this paper
• On the importance of initialization and momentum in deep learning

  2013cited by this paper
• Network inference with hidden units.

  2013cited by this paper
• Possible Principles Underlying the Transformations of Sensory Messages

  2012cited by this paper
• Improving neural networks by preventing co-adaptation of feature detectors

  2012cited by this paper
• On the difficulty of training recurrent neural networks

  2012cited by this paper
• Multimodal learning with deep Boltzmann machines

  2012cited by this paper
• ImageNet classification with deep convolutional neural networks

  2012cited by this paper
• Exact mean-field inference in asymmetric kinetic Ising systems

  2011cited by this paper
• Contractive Auto-Encoders: Explicit Invariance During Feature Extraction

  2011cited by this paper
• Facial Expression Transfer with Input-Output Temporal Restricted Boltzmann Machines

  2011cited by this paper
• Contextual tag inference

  2011cited by this paper
• Learning Recurrent Neural Networks with Hessian-Free Optimization

  2011cited by this paper
• Mean field theory for nonequilibrium network reconstruction.

  2010cited by this paper
• Fast Inference in Sparse Coding Algorithms with Applications to Object Recognition

  2010cited by this paper
• Dynamical binary latent variable models for 3D human pose tracking

  2010cited by this paper
• Why Does Unsupervised Pre-training Help Deep Learning?

  2010cited by this paper
• Network inference using asynchronously updated kinetic Ising model.

  2010cited by this paper
• Deep Boltzmann Machines

  2009cited by this paper
• The Recurrent Temporal Restricted Boltzmann Machine

  2008cited by this paper
• Extracting and composing robust features with denoising autoencoders

  2008cited by this paper
• Modeling Human Motion Using Binary Latent Variables

  2007cited by this paper
• Learning Multilevel Distributed Representations for High-Dimensional Sequences

  2007cited by this paper
• Sparse Feature Learning for Deep Belief Networks

  2007cited by this paper
• An empirical evaluation of deep architectures on problems with many factors of variation

  2007influential reference
• Making Working Memory Work: A Computational Model of Learning in the Prefrontal Cortex and Basal Ganglia

  2006cited by this paper
• Efficient Learning of Sparse Representations with an Energy-Based Model

  2006cited by this paper
• Modeling Human Motion Using Binary Latent Variables

  2006cited by this paper
• A Fast Learning Algorithm for Deep Belief Nets

  2006cited by this paper
• Localized activity profiles and storage capacity of rate-based autoassociative networks.

  2005cited by this paper
• Rate Models for Conductance-Based Cortical Neuronal Networks

  2003cited by this paper
• Training Products of Experts by Minimizing Contrastive Divergence

  2002cited by this paper
• Spatial structure and symmetry of simple-cell receptive fields in macaque primary visual cortex.

  2002cited by this paper
• Gradient Flow in Recurrent Nets: the Difficulty of Learning Long-Term Dependencies

  2001cited by this paper
• Efficient Learning in Boltzmann Machines Using Linear Response Theory

  1998cited by this paper
• Mean-field theory of Boltzmann machine learning

  1998cited by this paper
• Gradient-based learning applied to document recognition

  1998cited by this paper
• Long Short-Term Memory

  1997cited by this paper
• Sparse coding with an overcomplete basis set: a strategy employed by V1?

  1997cited by this paper
• The Helmholtz Machine

  1995cited by this paper
• The "wake-sleep" algorithm for unsupervised neural networks.

  1995cited by this paper
• The limitations of deterministic Boltzmann machine learning

  1993cited by this paper
• Are spin-glass effects relevant to understanding realistic auto-associative networks?

  1991cited by this paper
• Statistical mechanics for networks of graded-response neurons.

  1991cited by this paper
• Graded-response neurons and information encodings in autoassociative memories.

  1990cited by this paper
• Spin Glass Theory and Beyond

  1987cited by this paper
• Information processing in dynamical systems: foundations of harmony theory

  1986cited by this paper
• Learning representations of back-propagation errors

  1986cited by this paper
• Learning representations by back-propagating errors

  1986cited by this paper
• A Learning Algorithm for Boltzmann Machines

  1985cited by this paper
• Spin-glass models of neural networks.

  1985cited by this paper
• Neural networks and physical systems with emergent collective computational abilities.

  1982cited by this paper
• Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position

  1980cited by this paper
• Representation and recognition of the spatial organization of three-dimensional shapes

  1978cited by this paper
• Shape and arrangement of columns in cat's striate cortex

  1963cited by this paper
• Towards an information-flow model of human behaviour.

  1956cited by this paper
• Some informational aspects of visual perception.

  1954cited by this paper

• Artificial Intelligence and Neuroscience: Transformative Synergies in Brain Research and Clinical Applications

  2025cites this paper
• A Novel Neural Metric Based on Deep Boltzmann Machine

  2022cites this paper
• Quantifying Relevance in Learning and Inference

  2022cites this paper
• Restricted Boltzmann Machines as Models of Interacting Variables

  2021cites this paper
• A random energy approach to deep learning

  2021cites this paper
• Diversified Fisher kernel: encoding discrimination in Fisher features to compete deep neural models for visual classification task

  2020cites this paper
• A Molecular Image-Based Novel Quantitative Structure-Activity Relationship Approach, Deepsnap-Deep Learning and Machine Learning.

  2020cites this paper
• Transfer entropy-based feedback improves performance in artificial neural networks

  2017cites this paper
• Inverse Ising problem in continuous time: A latent variable approach.

  2017cites this paper
• A Novel Fractional Gradient-Based Learning Algorithm for Recurrent Neural Networks

  2017cites this paper
• Inverse statistical problems: from the inverse Ising problem to data science

  2017cites this paper
• Hybrid Network Model for “Deep Learning” of Chemical Data: Application to Antimicrobial Peptides

  2017cites this paper
• Correlation-compressed direct-coupling analysis

  2017cites this paper
• Improved pseudolikelihood regularization and decimation methods on non-linearly interacting systems with continuous variables

  2017cites this paper
• Toward an Integration of Deep Learning and Neuroscience

  2016cites this paper
• Sparse Neural Network Models of Antimicrobial Peptide‐Activity Relationships

  2016cites this paper
• Deep Learning in Drug Discovery

  2016cites this paper