An exact mapping between the Variational Renormalization Group and Deep Learning

Deep learning is a broad set of techniques that uses multiple layers of representation to automatically learn relevant features directly from structured data. Recently, such techniques have yielded record-breaking results on a diverse set of difficult machine learning tasks in computer vision, speech recognition, and natural language processing. Despite the enormous success of deep learning, relatively little is understood theoretically about why these techniques are so successful at feature learning and compression. Here, we show that deep learning is intimately related to one of the most important and successful techniques in theoretical physics, the renormalization group (RG). RG is an iterative coarse-graining scheme that allows for the extraction of relevant features (i.e. operators) as a physical system is examined at different length scales. We construct an exact mapping from the variational renormalization group, first introduced by Kadanoff, and deep learning architectures based on Restricted Boltzmann Machines (RBMs). We illustrate these ideas using the nearest-neighbor Ising Model in one and two-dimensions. Our results suggests that deep learning algorithms may be employing a generalized RG-like scheme to learn relevant features from data.

• Publication year

  2014

• Venue

  arXiv.org

• Publication date

  2014-10-14

• Fields of study

  Mathematics, Physics, Computer Science

• Identifiers
  arXiv 1410.3831
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• The results suggest that deep learning algorithms may employ a generalized renormalization-group-like scheme to learn relevant features from data.

  Confidence 0.80

  Anonymous (b2adb6bfad) extraction
• The nearest-neighbor Ising model in one and two dimensions is used to illustrate the mapping between variational renormalization group and deep learning.

  Confidence 0.95

  Anonymous (b2adb6bfad) extraction
• An exact mapping between the variational renormalization group and deep learning architectures based on Restricted Boltzmann Machines is constructed.

  Confidence 0.99

  Unknown

• deep learning

  method family

  A set of machine learning techniques that use multiple layers of representation to learn relevant features directly from structured data.

  Anonymous (b2adb6bfad) extraction
• feature learning

  learning process

  The process of automatically learning relevant features from data, which deep learning is described as performing in this paper.

  Anonymous (b2adb6bfad) extraction
• nearest-neighbor ising model

  physical system, example system

  An Ising model with nearest-neighbor interactions that is used here in one and two dimensions as an example system.

  Aliases: Ising Model

  Anonymous (b2adb6bfad) extraction
• renormalization group

  theoretical framework, method

  An iterative coarse-graining scheme for extracting relevant features or operators as a physical system is viewed at different length scales.

  Aliases: RG

  Anonymous (b2adb6bfad) extraction
• restricted boltzmann machines

  model, method

  A class of deep learning architectures used in this paper to establish the mapping to variational renormalization group.

  Aliases: RBMs, RBM

  Anonymous (b2adb6bfad) extraction
• variational renormalization group

  theoretical framework, method

  A variational form of the renormalization group introduced by Kadanoff and used here as one side of the exact mapping.

  Anonymous (b2adb6bfad) extraction

• Representation Learning: A Review and New Perspectives

  2012cited by this paper
• Advances in Neural Information Processing Systems 25

  2012cited by this paper
• Optimization with Sparsity-Inducing Penalties (Foundations and Trends(R) in Machine Learning)

  2011cited by this paper
• Electron – molecule collision calculations using the R-matrix method

  2010cited by this paper
• Proceedings of the 25th international conference on Machine learning

  2008cited by this paper
• Large-scale kernel machines

  2007cited by this paper
• Proceedings of the 24th international conference on Machine learning

  2007cited by this paper
• on Pattern Analysis and Machine Intelligence

  2005cited by this paper
• Reviews of Modern Physics

  2002cited by this paper
• In Advances in Neural Information Processing Systems

  1996cited by this paper
• Scaling and Renormalization in Statistical Physics: Mean field theory

  1996cited by this paper
• Neural Computation

  1989cited by this paper
• and as an in

  year unknowncited by this paper
• IEEE Transactions on Audio, Speech, and Language Processing

  year unknowncited by this paper
• Foundations and Trends R (cid:13) in Technology, Information and Operations Management Cash Beer Game

  year unknowncited by this paper

• Hierarchical Zero-Order Optimization for Deep Neural Networks

  2026cites this paper
• Machine Learning the Strong Disorder Renormalization Group Method for Disordered Quantum Spin Chains

  2026cites this paper
• Detecting nonequilibrium phase transitions via continuous monitoring of space-time trajectories and autoencoder-based clustering

  2026cites this paper
• Unsupervised Discovery of Intermediate Phase Order in the Frustrated $J_1$-$J_2$ Heisenberg Model via Prometheus Framework

  2026cites this paper
• Latent Object Permanence: Topological Phase Transitions, Free-Energy Principles, and Renormalization Group Flows in Deep Transformer Manifolds

  2026cites this paper
• Towards Worst-Case Guarantees with Scale-Aware Interpretability

  2026cites this paper
• Deep Learning of Compositional Targets with Hierarchical Spectral Methods

  2026cites this paper
• Unsupervised Ensemble Learning Through Deep Energy-based Models

  2026cites this paper
• Interpreting deep learning by establishing a rigorous corresponding relationship with the renormalization group on the Ising model

  2025influential citation
• How compositional generalization and creativity improve as diffusion models are trained

  2025cites this paper
• A Two-Phase Perspective on Deep Learning Dynamics

  2025cites this paper
• Emergent weight morphologies in deep neural networks

  2025cites this paper
• Symmetry and Generalisation in Neural Approximations of Renormalisation Transformations

  2025cites this paper
• Bulk-boundary decomposition of neural networks

  2025cites this paper
• Quantum Geometry insights in Deep Learning

  2025cites this paper
• RGMem: Renormalization Group-inspired Memory Evolution for Language Agents

  2025cites this paper
• Group Convolutional Neural Network Ground State of the Quantum Dimer Model

  2025cites this paper
• Generalization Dynamics of Linear Diffusion Models

  2025cites this paper
• Functional Renormalization for Signal Detection: Dimensional Analysis and Dimensional Phase Transition for Nearly Continuous Spectra Effective Field Theory

  2025cites this paper
• Optimizing Latent Dimension Allocation in Hierarchical VAEs: Balancing Attenuation and Information Retention for OOD Detection

  2025cites this paper
• Data augmentation using diffusion models to enhance inverse Ising inference

  2025cites this paper
• Cognition all the way down 2.0: neuroscience beyond neurons in the diverse intelligence era

  2025cites this paper
• On the Road to Clarity: Exploring Explainable AI for World Models in a Driver Assistance System

  2024cites this paper
• Computational experiments with cellular-automata generated images reveal intrinsic limitations of convolutional neural networks on pattern recognition tasks

  2024cites this paper
• Magic Class and the Convolution Group

  2024cites this paper
• Learning phase transitions by siamese neural network

  2024cites this paper
• MS3D: A RG Flow-Based Regularization for GAN Training with Limited Data

  2024cites this paper
• Replica symmetry breaking in supervised and unsupervised Hebbian networks

  2024cites this paper
• Internal Representations in Spiking Neural Networks, criticality and the Renormalization Group

  2024cites this paper
• Absolute abstraction: a renormalisation group approach

  2024cites this paper
• Mind the information gap: How sampling and clustering impact the predictability of reach‐scale channel types in California (USA)

  2024cites this paper
• Dynamic neuron approach to deep neural networks: Decoupling neurons for renormalization group analysis

  2024cites this paper
• A Statistical Physics Perspective: Understanding the Causality Behind Convolutional Neural Network Adversarial Vulnerability

  2024cites this paper
• On the Arrow of Inference

  2024cites this paper
• Renormalization group flow, optimal transport, and diffusion-based generative model.

  2024cites this paper
• Wilsonian renormalization of neural network Gaussian processes

  2024cites this paper
• Multi-scale Simulation of Complex Systems: A Perspective of Integrating Knowledge and Data

  2023cites this paper
• Topos of Noise

  2023cites this paper
• Deep Learning for Dialogue Systems: Chit-Chat and Beyond

  2023cites this paper
• Multi-Relevance: Coexisting but Distinct Notions of Scale in Large Systems

  2023cites this paper
• Bayesian renormalization

  2023cites this paper
• Lightweight ECG signal classification via linear law-based feature extraction

  2023cites this paper
• Renormalization Group-Motivated Learning

  2023cites this paper
• A functional renormalization group for signal detection and stochastic ergodicity breaking

  2023cites this paper
• Renormalizing Diffusion Models

  2023cites this paper
• Unsupervised and Supervised learning by Dense Associative Memory under replica symmetry breaking

  2023cites this paper
• Quantum Classical Algorithm for the Study of Phase Transitions in the Hubbard Model via Dynamical Mean-Field Theory

  2023cites this paper
• Iterative Magnitude Pruning as a Renormalisation Group: A Study in The Context of The Lottery Ticket Hypothesis

  2023cites this paper
• Study of phase transition of Potts model with Domain Adversarial Neural Network

  2022cites this paper
• Inferring Cultural Landscapes with the Inverse Ising Model

  2022cites this paper
• Renormalization in the neural network-quantum field theory correspondence

  2022cites this paper
• Quantum Reservoir Computing Implementations for Classical and Quantum Problems

  2022cites this paper
• Dense Hebbian Neural Networks: A Replica Symmetric Picture of Supervised Learning

  2022cites this paper
• Interpreting Deep Learning by Establishing a Rigorous Corresponding Relationship with Renormalization Group

  2022cites this paper
• Thermodynamics of the Ising Model Encoded in Restricted Boltzmann Machines

  2022cites this paper
• How critical is brain criticality?

  2022cites this paper
• Lateral predictive coding revisited: internal model, symmetry breaking, and response time

  2022cites this paper
• Learning the black hole metric from holographic conductivity

  2022cites this paper
• Entangling Solid Solutions: Machine Learning of Tensor Networks for Materials Property Prediction

  2022cites this paper
• Complexity from Adaptive-Symmetries Breaking: Global Minima in the Statistical Mechanics of Deep Neural Networks

  2022cites this paper
• Renormalization group flow as optimal transport

  2022cites this paper
• Entropy of Artificial Intelligence

  2022cites this paper
• Categorical representation learning and RG flow operators for algorithmic classifiers

  2022cites this paper
• Emergent quantum mechanics at the boundary of a local classical lattice model

  2022cites this paper
• Wavelet Conditional Renormalization Group

  2022cites this paper
• Research Analysis on Multi Representation in Physical Materials in The Year of 2014 to 2021

  2022cites this paper
• Combining Fractional Derivatives and Machine Learning: A Review

  2022cites this paper
• Architecture representations for quantum convolutional neural networks

  2022cites this paper
• Field theoretical approach for signal detection in nearly continuous positive spectra III: Universal features

  2022cites this paper
• Flow of Information in Hopfield Neural Networks

  2022cites this paper
• Engineering flexible machine learning systems by traversing functionally invariant paths

  2022cites this paper
• Misanthropic Entropy and Renormalization as a Communication Channel

  2021cites this paper
• Non-perturbative renormalization for the neural network-QFT correspondence

  2021cites this paper
• Machine learning study of the deformed one-dimensional topological superconductor

  2021cites this paper
• Hybrid Classical-Quantum Approaches for Anomaly Detection

  2021cites this paper
• Physics-informed machine learning

  2021cites this paper
• An Enquiry on Similarities between Renormalization Group and Auto-Encoders Using Transfer Learning

  2021influential citation
• Entanglement transitions from restricted Boltzmann machines

  2021cites this paper
• Entropy regularized reinforcement learning using large deviation theory

  2021cites this paper
• Efficient modeling of trivializing maps for lattice ϕ4 theory using normalizing flows: A first look at scalability

  2021cites this paper
• Presence and Absence of Barren Plateaus in Tensor-Network Based Machine Learning.

  2021cites this paper
• Towards quantifying information flows: relative entropy in deep neural networks and the renormalization group

  2021cites this paper
• Inverse renormalization group based on image super-resolution using deep convolutional networks

  2021cites this paper
• Neural networks in quantum many-body physics: a hands-on tutorial

  2021cites this paper
• Tensor networks and efficient descriptions of classical data

  2021cites this paper
• Pulmonary Functional Imaging: Basics and Clinical Applications

  2021cites this paper
• Linear Transformations in Autoencoder Latent Space Predict Time Translations in Active Matter System

  2021cites this paper
• Universality of Winning Tickets: A Renormalization Group Perspective

  2021cites this paper
• Feature extraction of machine learning and phase transition point of Ising model

  2021cites this paper
• Machine learning of pair-contact process with diffusion

  2021cites this paper
• A New Triangle: Fractional Calculus, Renormalization Group, and Machine Learning

  2021influential citation
• The Physics of Machine Learning: An Intuitive Introduction for the Physical Scientist

  2021cites this paper
• Disorder averaging and its UV discontents

  2021cites this paper
• Applying quantum approximate optimization to the heterogeneous vehicle routing problem

  2021cites this paper
• The Multiple Dimensions of Networks in Cancer: A Perspective

  2021cites this paper
• The Autodidactic Universe

  2021cites this paper
• How To Use Neural Networks To Investigate Quantum Many-Body Physics

  2021cites this paper
• Super-resolution of spin configurations based on flow-based generative models

  2021cites this paper
• Machine learning for quantum matter

  2020cites this paper
• Generalized scale behavior and renormalization group for principal component analysis

  2020cites this paper