Learning Using a Single Forward Pass

We propose a learning algorithm to overcome the limitations of traditional backpropagation in resource-constrained environments: Solo Pass Embedded Learning Algorithm (SPELA). SPELA operates with local loss functions to update weights, significantly saving on resources allocated to the propagation of gradients and storing computational graphs while being sufficiently accurate. Consequently, SPELA can closely match backpropagation using less memory. Moreover, SPELA can effectively fine-tune pre-trained image recognition models for new tasks. Further, SPELA is extended with significant modifications to train CNN networks, which we evaluate on CIFAR-10, CIFAR-100, and SVHN 10 datasets, showing equivalent performance compared to backpropagation. Our results indicate that SPELA, with its features such as local learning and early exit, is a potential candidate for learning in resource-constrained edge AI applications.

• Publication year

  2024

• Venue

  Trans. Mach. Learn. Res.

• Publication date

  2024-02-15

• Fields of study

  Computer Science

• Identifiers
  arXiv 2402.09769
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Dendritic Localized Learning: Toward Biologically Plausible Algorithm

  2025cited by this paper
• NoProp: Training Neural Networks without Full Back-propagation or Full Forward-propagation

  2025cited by this paper
• Self-Contrastive Forward-Forward algorithm

  2024cited by this paper
• Forward Learning of Graph Neural Networks

  2024cited by this paper
• Theoretical Analysis of Inductive Biases in Deep Convolutional Networks

  2023cited by this paper
• SymBa: Symmetric Backpropagation-Free Contrastive Learning with Forward-Forward Algorithm for Optimizing Convergence

  2023cited by this paper
• Backpropagation-free training of deep physical neural networks

  2023cited by this paper
• The Trifecta: Three simple techniques for training deeper Forward-Forward networks

  2023cited by this paper
• Investigating Random Variations of the Forward-Forward Algorithm for Training Neural Networks

  2023cited by this paper
• Suitability of Forward-Forward and PEPITA Learning to MLCommons-Tiny benchmarks

  2023cited by this paper
• On-device Training: A First Overview on Existing Systems

  2022cited by this paper
• Error-driven Input Modulation: Solving the Credit Assignment Problem without a Backward Pass

  2022cited by this paper
• Why Should We Add Early Exits to Neural Networks?

  2020cited by this paper
• Inductive biases for deep learning of higher-level cognition

  2020cited by this paper
• TinyTL: Reduce Memory, Not Parameters for Efficient On-Device Learning

  2020influential reference
• Local plasticity rules can learn deep representations using self-supervised contrastive predictions

  2020cited by this paper
• Theories of Error Back-Propagation in the Brain

  2019cited by this paper
• A Spiking Neural Network with Local Learning Rules Derived from Nonnegative Similarity Matching

  2019cited by this paper
• Learning to solve the credit assignment problem

  2019cited by this paper
• Deep Learning without Weight Transport

  2019cited by this paper
• Universal Sentence Encoder

  2018cited by this paper
• Understanding Synthetic Gradients and Decoupled Neural Interfaces

  2017cited by this paper
• The High-Dimensional Geometry of Binary Neural Networks

  2017cited by this paper
• Decoupled Neural Interfaces using Synthetic Gradients

  2016cited by this paper
• Deep Learning

  2016cited by this paper
• How Important Is Weight Symmetry in Backpropagation?

  2015cited by this paper
• Random feedback weights support learning in deep neural networks

  2014cited by this paper
• Universally optimal distribution of points on spheres

  2006cited by this paper
• Distributing many points on a sphere

  1997cited by this paper
• Backpropagation Through Time: What It Does and How to Do It

  1990cited by this paper
• Backpropagation Applied to Handwritten Zip Code Recognition

  1989cited by this paper
• Learning representations by back-propagating errors

  1986cited by this paper

• Learning without Global Backpropagation via Synergistic Information Distillation

  2025cites this paper