Forward Learning with Top-Down Feedback: Empirical and Analytical Characterization

"Forward-only"algorithms, which train neural networks while avoiding a backward pass, have recently gained attention as a way of solving the biologically unrealistic aspects of backpropagation. Here, we first address compelling challenges related to the"forward-only"rules, which include reducing the performance gap with backpropagation and providing an analytical understanding of their dynamics. To this end, we show that the forward-only algorithm with top-down feedback is well-approximated by an"adaptive-feedback-alignment"algorithm, and we analytically track its performance during learning in a prototype high-dimensional setting. Then, we compare different versions of forward-only algorithms, focusing on the Forward-Forward and PEPITA frameworks, and we show that they share the same learning principles. Overall, our work unveils the connections between three key neuro-inspired learning rules, providing a link between"forward-only"algorithms, i.e., Forward-Forward and PEPITA, and an approximation of backpropagation, i.e., Feedback Alignment.

• Publication year

  2023

• Venue

  International Conference on Learning Representations

• Publication date

  2023-02-10

• Fields of study

  Computer Science

• Identifiers
  DOI 10.48550/arXiv.2302.05440arXiv 2302.05440
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• The Predictive Forward-Forward Algorithm

  2023cited by this paper
• The combination of Hebbian and predictive plasticity learns invariant object representations in deep sensory networks

  2023cited by this paper
• Layer Collaboration in the Forward-Forward Algorithm

  2023cited by this paper
• The Influence of Learning Rule on Representation Dynamics in Wide Neural Networks

  2022cited by this paper
• Spike-based local synaptic plasticity: a survey of computational models and neuromorphic circuits

  2022cited by this paper
• Hebbian Deep Learning Without Feedback

  2022cited by this paper
• Introducing principles of synaptic integration in the optimization of deep neural networks

  2022cited by this paper
• Error-driven Input Modulation: Solving the Credit Assignment Problem without a Backward Pass

  2022influential reference
• Phase diagram of stochastic gradient descent in high-dimensional two-layer neural networks

  2022cited by this paper
• The Forward-Forward Algorithm: Some Preliminary Investigations

  2022influential reference
• Convolutional Neural Generative Coding: Scaling Predictive Coding to Natural Images

  2022cited by this paper
• Learning Without Feedback: Fixed Random Learning Signals Allow for Feedforward Training of Deep Neural Networks

  2021influential reference
• Online Training of Spiking Recurrent Neural Networks with Phase-Change Memory Synapses

  2021influential reference
• Credit Assignment in Neural Networks through Deep Feedback Control

  2021cited by this paper
• Align, then memorise: the dynamics of learning with feedback alignment

  2021influential reference
• Credit Assignment Through Broadcasting a Global Error Vector

  2021cited by this paper
• Predictive Coding Can Do Exact Backpropagation on Convolutional and Recurrent Neural Networks

  2021cited by this paper
• Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeed

  2021cited by this paper
• Biological and Computer Vision

  2021cited by this paper
• Training End-to-End Analog Neural Networks with Equilibrium Propagation

  2020cited by this paper
• Backpropagation and the brain

  2020cited by this paper
• Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architectures

  2020cited by this paper
• Large-Scale Gradient-Free Deep Learning with Recursive Local Representation Alignment.

  2020cited by this paper
• The neural coding framework for learning generative models

  2020cited by this paper
• Local plasticity rules can learn deep representations using self-supervised contrastive predictions

  2020cited by this paper
• Deep Learning without Weight Transport

  2019cited by this paper
• Training Neural Networks with Local Error Signals

  2019cited by this paper
• Modeling the Influence of Data Structure on Learning in Neural Networks: The Hidden Manifold Model

  2019cited by this paper
• A deep learning framework for neuroscience

  2019cited by this paper
• Dynamics of stochastic gradient descent for two-layer neural networks in the teacher–student setup

  2019cited by this paper
• Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architectures

  2018cited by this paper
• Contrastive Hebbian Learning with Random Feedback Weights

  2018cited by this paper
• Biologically Motivated Algorithms for Propagating Local Target Representations

  2018cited by this paper
• Unsupervised Learning with Self-Organizing Spiking Neural Networks

  2018cited by this paper
• Error Forward-Propagation: Reusing Feedforward Connections to Propagate Errors in Deep Learning

  2018cited by this paper
• Learning in the machine: Recirculation is random backpropagation

  2018cited by this paper
• Biologically-plausible learning algorithms can scale to large datasets

  2018cited by this paper
• Feedback alignment in deep convolutional networks

  2018cited by this paper
• Representation Learning with Contrastive Predictive Coding

  2018cited by this paper
• Control of synaptic plasticity in deep cortical networks

  2018cited by this paper
• Random synaptic feedback weights support error backpropagation for deep learning

  2016influential reference
• Equilibrium Propagation: Bridging the Gap between Energy-Based Models and Backpropagation

  2016cited by this paper
• Direct Feedback Alignment Provides Learning in Deep Neural Networks

  2016influential reference
• Statistical physics of inference: thresholds and algorithms

  2015cited by this paper
• How Important Is Weight Symmetry in Backpropagation?

  2015cited by this paper
• Difference Target Propagation

  2014cited by this paper
• A History of Spike-Timing-Dependent Plasticity

  2011cited by this paper
• The conference paper

  2011cited by this paper
• Real-Time Classification of Complex Patterns Using Spike-Based Learning in Neuromorphic VLSI

  2009cited by this paper
• Spike timing-dependent plasticity: a Hebbian learning rule.

  2008cited by this paper
• Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning

  2004cited by this paper
• Equivalence of Backpropagation and Contrastive Hebbian Learning in a Layered Network

  2003cited by this paper
• Rate, timing, and cooperativity jointly determine cortical synaptic plasticity.

  2001cited by this paper
• Statistical Mechanics of Learning

  2001cited by this paper
• Crafting Papers on Machine Learning

  2000cited by this paper
• Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects.

  1999cited by this paper
• On-line learning in soft committee machines.

  1995cited by this paper
• Backpropagation: the basic theory

  1995cited by this paper
• On-line backpropagation in two-layered neural networks

  1995cited by this paper
• Learning by on-line gradient descent

  1995cited by this paper
• Exact solution for on-line learning in multilayer neural networks.

  1995cited by this paper
• THE STATISTICAL-MECHANICS OF LEARNING A RULE

  1993cited by this paper
• Statistical mechanics of learning from examples.

  1992cited by this paper
• A more biologically plausible learning rule for neural networks.

  1991cited by this paper
• A Simple Weight Decay Can Improve Generalization

  1991cited by this paper
• Three unfinished works on the optimal storage capacity of networks

  1989cited by this paper
• Learning Representations by Recirculation

  1987cited by this paper

• Can Local Learning Match Self-Supervised Backpropagation?

  2026cites this paper
• Closed-Form Feedback-Free Learning with Forward Projection

  2025cites this paper
• Forward Only Learning for Orthogonal Neural Networks of any Depth

  2025influential citation
• A statistical physics framework for optimal learning

  2025cites this paper
• Learning Online MEMS Calibration with Time-Varying and Memory-Efficient Gaussian Neural Topologies

  2025cites this paper
• Online On-Device Adaptation of Linguistic Fuzzy Models for TinyML Systems

  2025cites this paper
• Optimal protocols for continual learning via statistical physics and control theory

  2024cites this paper
• Beyond Backpropagation: Optimization with Multi-Tangent Forward Gradients

  2024cites this paper
• Exploring the Forward-Forward Algorithm to Train Neural Networks for Camera Trap Images on the Edge

  2024cites this paper
• SHELOB-FFL: addressing Systems HEterogeneity with LOcally Backpropagated Forward-Forward Learning

  2024cites this paper
• Prediction Accuracy and Adversarial Robustness of Error-Based Input Perturbation Learning

  2024influential citation
• Lightweight Inference for Forward-Forward Algorithm

  2024cites this paper
• Learning Pressure Sensor Drifts From Zero Deployability Budget

  2024cites this paper
• In-Sensor Learning for Pressure Self-Calibration

  2024cites this paper
• Self-Contrastive Forward-Forward algorithm

  2024influential citation
• Intrinsic Biologically Plausible Adversarial Robustness

  2023cites this paper
• Suitability of Forward-Forward and PEPITA Learning to MLCommons-Tiny benchmarks

  2023cites this paper
• Backpropagation-free training of deep physical neural networks

  2023cites this paper
• Biologically Plausible Training of Deep Neural Networks Using a Top-down Credit Assignment Network

  2022cites this paper
• Intrinsic Biologically Plausible Adversarial Training

  year unknowninfluential citation
• Lightweight Inference for Forward-Forward Training Algorithm

  year unknowncites this paper