Emergence of Compositional Representations in Restricted Boltzmann Machines

Extracting automatically the complex set of features composing real high-dimensional data is crucial for achieving high performance in machine-learning tasks. Restricted Boltzmann machines (RBM) are empirically known to be efficient for this purpose, and to be able to generate distributed and graded representations of the data. We characterize the structural conditions (sparsity of the weights, low effective temperature, nonlinearities in the activation functions of hidden units, and adaptation of fields maintaining the activity in the visible layer) allowing RBM to operate in such a compositional phase. Evidence is provided by the replica analysis of an adequate statistical ensemble of random RBMs and by RBM trained on the handwritten digits data set MNIST.

• Publication year

  2016

• Venue

  Physical Review Letters

• Publication date

  2016-11-21

• Fields of study

  Mathematics, Physics, Computer Science, Medicine

• Identifiers
  DOI 10.1103/PhysRevLett.118.138301arXiv 1611.06759PMID 28409983
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar, PubMed

• No claims are published for this paper.

• No concepts are published for this paper.

• Theano: A Python framework for fast computation of mathematical expressions

  2016cited by this paper
• Extensive parallel processing on scale-free networks.

  2014cited by this paper
• Representation Learning: A Review and New Perspectives

  2012cited by this paper
• Multitasking associative networks.

  2011cited by this paper
• On the equivalence of Hopfield networks and Boltzmann Machines

  2011cited by this paper
• Proceedings of The 28th International Conference on Machine Learning

  2011cited by this paper
• Proceedings of the 25th international conference on Machine learning

  2008cited by this paper
• Proceedings of the 24th international conference on Machine learning

  2007influential reference
• The mnist database of handwritten digits

  2005cited by this paper
• IEEE Transactions on Pattern Analysis and Machine Intelligence

  2004cited by this paper
• In Advances in Neural Information Processing Systems

  1996cited by this paper
• Emergence of simple-cell receptive field properties by learning a sparse code for natural images

  1996cited by this paper
• Quantitative estimate of the information relayed by the Schaffer collaterals

  1995cited by this paper
• Neural networks

  1995cited by this paper
• Parallel Distributed Processing: Explorations in the Microstructures of Cognition

  1987cited by this paper
• Information processing in dynamical systems: foundations of harmony theory

  1986cited by this paper
• Storing infinite numbers of patterns in a spin-glass model of neural networks.

  1985cited by this paper
• Neural networks and physical systems with emergent collective computational abilities.

  1982cited by this paper

• Serial vs parallel recall in the Blume-Every-Griffiths neural networks

  2026cites this paper
• On the supra-linear storage in dense networks of grid and place cells

  2025cites this paper
• Spin glasses and the Parisi formula

  2025cites this paper
• Fast nonlinear integration drives accurate encoding of input information in large multiscale systems

  2025cites this paper
• Emergence of Nonequilibrium Latent Cycles in Unsupervised Generative Modeling

  2025cites this paper
• The beneficial role of noises for disentanglement tasks in modular Hebbian networks

  2025cites this paper
• On the role of non-linear latent features in bipartite generative neural networks

  2025influential citation
• Learning restricted Boltzmann machines with pattern induced weights

  2024cites this paper
• Fundamental operating regimes, hyper-parameter fine-tuning and glassiness: towards an interpretable replica-theory for trained restricted Boltzmann machines

  2024cites this paper
• Replica symmetry breaking in supervised and unsupervised Hebbian networks

  2024cites this paper
• The recurrent temporal restricted Boltzmann machine captures neural assembly dynamics in whole-brain activity

  2024influential citation
• GenEFT: Understanding statics and dynamics of model generalization via physics-inspired effective theory.

  2024cites this paper
• Replica method for computational problems with randomness: principles and illustrations

  2024cites this paper
• Modeling structured data learning with Restricted Boltzmann machines in the teacher-student setting

  2024influential citation
• An informal introduction to the Parisi formula

  2024cites this paper
• The thermodynamic limit in mean field neural networks

  2024cites this paper
• Generalized hetero-associative neural networks

  2024cites this paper
• Configuration interaction guided sampling with interpretable restricted Boltzmann machine

  2024cites this paper
• Compositionally Equivariant Representation Learning

  2023cites this paper
• Unsupervised and Supervised learning by Dense Associative Memory under replica symmetry breaking

  2023cites this paper
• Inferring effective couplings with Restricted Boltzmann Machines

  2023cites this paper
• Parallel learning by multitasking neural networks

  2023cites this paper
• Statistical mechanics of learning via reverberation in bidirectional associative memories

  2023influential citation
• Fast and Functional Structured Data Generators Rooted in Out-of-Equilibrium Physics

  2023cites this paper
• Transition paths in Potts-like energy landscapes: General properties and application to protein sequence models.

  2023cites this paper
• Storage and Learning Phase Transitions in the Random-Features Hopfield Model.

  2023cites this paper
• Learning Interacting Theories from Data

  2023cites this paper
• Interpretable Machine Learning of Amino Acid Patterns in Proteins: A Statistical Ensemble Approach

  2023cites this paper
• Unsupervised hierarchical clustering using the learning dynamics of RBMs

  2023cites this paper
• Funneling modulatory peptide design with generative models: Discovery and characterization of disruptors of calcineurin protein-protein interactions

  2023cites this paper
• Quantifying Relevance in Learning and Inference

  2022cites this paper
• Machine learning phases of an Abelian gauge theory

  2022cites this paper
• Storing, learning and retrieving biased patterns. (English)

  2022cites this paper
• A Developmental Approach for Training Deep Belief Networks

  2022cites this paper
• Disentangling representations in Restricted Boltzmann Machines without adversaries

  2022cites this paper
• Replica analysis of the lattice-gas restricted Boltzmann machine partition function

  2022cites this paper
• Storing, learning and retrieving biased patterns

  2022cites this paper
• Learning a Restricted Boltzmann Machine using biased Monte Carlo sampling

  2022cites this paper
• Pattern reconstruction with restricted Boltzmann machines

  2022cites this paper
• Generative and interpretable machine learning for aptamer design and analysis of in vitro sequence selection

  2022cites this paper
• Fundamental problems in Statistical Physics XIV: Lecture on Machine Learning

  2022cites this paper
• Exact Training of Restricted Boltzmann Machines on Intrinsically Low Dimensional Data.

  2021cites this paper
• Restricted Boltzmann Machines as Models of Interacting Variables

  2021cites this paper
• Synchronization of Stochastic Memristive Neural Networks with Retarded and Advanced Argument

  2021cites this paper
• Using top-down modulation to optimally balance shared versus separated task representations

  2021cites this paper
• CD8+ T-cell Epitope Presentation on MHC-1 as a Probability Space

  2021cites this paper
• On the effective initialisation for restricted Boltzmann machines via duality with Hopfield model

  2021cites this paper
• The emergence of a concept in shallow neural networks

  2021cites this paper
• Learning and Retrieval Operational Modes for Three-Layer Restricted Boltzmann Machines

  2021cites this paper
• A universal neural network for learning phases

  2021cites this paper
• Neural assemblies uncovered by generative modeling explain whole-brain activity statistics and reflect structural connectivity

  2021influential citation
• Variational mean-field theory for training restricted Boltzmann machines with binary synapses.

  2020cites this paper
• Probing T-cell response by sequence-based probabilistic modeling

  2020cites this paper
• RBM-MHC: A Semi-Supervised Machine-Learning Method for Sample-Specific Prediction of Antigen Presentation by HLA-I Alleles

  2020cites this paper
• Flexible machine learning prediction of antigen presentation for rare and common HLA-I alleles

  2020cites this paper
• Tensor Network States, Entanglement, and Anomalies of Topological Phases of Matters

  2020influential citation
• Understanding collective behaviors in reinforcement learning evolutionary games via a belief-based formalization.

  2020cites this paper
• Generating Digital Twins with Multiple Sclerosis Using Probabilistic Neural Networks

  2020cites this paper
• Restricted Boltzmann machine: Recent advances and mean-field theory

  2020cites this paper
• Replica symmetry breaking in neural networks: a few steps toward rigorous results

  2020cites this paper
• A dynamical mean-field theory for learning in restricted Boltzmann machines

  2020cites this paper
• Boltzmann Machines as Generalized Hopfield Networks: A Review of Recent Results and Outlooks

  2020cites this paper
• Maximal relevance and optimal learning machines

  2019cites this paper
• Generalized Guerra's interpolation schemes for dense associative neural networks

  2019cites this paper
• Oscillatory evolution of collective behavior in evolutionary games played with reinforcement learning

  2019cites this paper
• Mean-field inference methods for neural networks

  2019cites this paper
• Gaussian-spherical restricted Boltzmann machines

  2019cites this paper
• Machine learning and the physical sciences

  2019cites this paper
• Learning Compositional Representations of Interacting Systems with Restricted Boltzmann Machines: Comparative Study of Lattice Proteins

  2019influential citation
• Natural representation of composite data with replicated autoencoders

  2019cites this paper
• On the Marchenko–Pastur law in analog bipartite spin-glasses

  2019influential citation
• Minimal model of permutation symmetry in unsupervised learning

  2019cites this paper
• ‘Place-cell’ emergence and learning of invariant data with restricted Boltzmann machines: breaking and dynamical restoration of continuous symmetries in the weight space

  2019cites this paper
• Self-regularizing restricted Boltzmann machines

  2019cites this paper
• Machine learning for comprehensive forecasting of Alzheimer’s Disease progression

  2018cites this paper
• Deep Learning in Spiking Neural Networks

  2018cites this paper
• Neuro-heuristics for nonlinear singular Thomas-Fermi systems

  2018cites this paper
• Learning protein constitutive motifs from sequence data

  2018cites this paper
• A high-bias, low-variance introduction to Machine Learning for physicists

  2018cites this paper
• Restricted Boltzmann machines and matrix product states of one-dimensional translationally invariant stabilizer codes

  2018cites this paper
• A novel derivation of the Marchenko-Pastur law through analog bipartite spin-glasses

  2018influential citation
• TherelativisticHopfield network: Rigorous results

  2018cites this paper
• Free energies of Boltzmann machines: self-averaging, annealed and replica symmetric approximations in the thermodynamic limit

  2018cites this paper
• Dreaming neural networks: rigorous results

  2018cites this paper
• Non-convex Multi-species Hopfield Models

  2018cites this paper
• Mechanisms of dimensionality reduction and decorrelation in deep neural networks

  2018cites this paper
• Thermodynamics of Restricted Boltzmann Machines and Related Learning Dynamics

  2018influential citation
• Dreaming neural networks: forgetting spurious memories and reinforcing pure ones

  2018cites this paper
• A new mechanical approach to handle generalized Hopfield neural networks

  2018cites this paper
• Analytic continuation via domain knowledge free machine learning

  2018cites this paper
• Decreasing the Size of the Restricted Boltzmann Machine

  2018cites this paper
• Approximate survey propagation for statistical inference

  2018cites this paper
• A relativistic extension of Hopfield neural networks via the mechanical analogy

  2018cites this paper
• Generalized pinning synchronization of delayed Cohen-Grossberg neural networks with discontinuous activations

  2018cites this paper
• Role of zero synapses in unsupervised feature learning

  2017cites this paper
• Variational Probability Flow for Biologically Plausible Training of Deep Neural Networks

  2017cites this paper
• Information Perspective to Probabilistic Modeling: Boltzmann Machines versus Born Machines

  2017cites this paper
• Neural Networks Retrieving Boolean Patterns in a Sea of Gaussian Ones

  2017cites this paper
• Constrained low-rank matrix estimation: phase transitions, approximate message passing and applications

  2017cites this paper
• A Deterministic and Generalized Framework for Unsupervised Learning with Restricted Boltzmann Machines

  2017cites this paper