How Important Is Weight Symmetry in Backpropagation?

Gradient backpropagation (BP) requires symmetric feedforward and feedback connections — the same weights must be used for forward and backward passes. This "weight transport problem'' (Grossberg 1987) is thought to be one of the main reasons to doubt BP's biologically plausibility. Using 15 different classification datasets, we systematically investigate to what extent BP really depends on weight symmetry. In a study that turned out to be surprisingly similar in spirit to Lillicrap et al.'s demonstration (Lillicrap et al. 2014) but orthogonal in its results, our experiments indicate that: (1) the magnitudes of feedback weights do not matter to performance (2) the signs of feedback weights do matter — the more concordant signs between feedforward and their corresponding feedback connections, the better (3) with feedback weights having random magnitudes and 100% concordant signs, we were able to achieve the same or even better performance than SGD. (4) some normalizations/stabilizations are indispensable for such asymmetric BP to work, namely Batch Normalization (BN) (Ioffe and Szegedy 2015) and/or a "Batch Manhattan'' (BM) update rule.

• Publication year

  2015

• Venue

  AAAI Conference on Artificial Intelligence

• Publication date

  2015-10-17

• Fields of study

  Computer Science

• Identifiers
  DOI 10.1609/aaai.v30i1.10279arXiv 1510.05067
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Top Downloads in IEEE Xplore [Reader's Choice]

  2017cited by this paper
• Multi-modal human aggression detection

  2016cited by this paper
• Towards Biologically Plausible Deep Learning

  2015cited by this paper
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

  2015cited by this paper
• Manhattan rule training for memristive crossbar circuit pattern classifiers

  2015cited by this paper
• How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagation

  2014cited by this paper
• Subtasks of Unconstrained Face Recognition

  2014cited by this paper
• MatConvNet: Convolutional Neural Networks for MATLAB

  2014influential reference
• DeepFace: Closing the Gap to Human-Level Performance in Face Verification

  2014cited by this paper
• Neural Turing Machines

  2014cited by this paper
• iCub World: Friendly Robots Help Building Good Vision Data-Sets

  2013influential reference
• Distributed Representations of Words and Phrases and their Compositionality

  2013cited by this paper
• ImageNet classification with deep convolutional neural networks

  2012cited by this paper
• Applying Convolutional Neural Networks concepts to hybrid NN-HMM model for speech recognition

  2012cited by this paper
• Phone Recognition on the TIMIT Database

  2012cited by this paper
• Adaptive Optimal Control Without Weight Transport

  2012cited by this paper
• Reading Digits in Natural Images with Unsupervised Feature Learning

  2011cited by this paper
• Scaling up biologically-inspired computer vision: A case study in unconstrained face recognition on facebook

  2011cited by this paper
• An Analysis of Single-Layer Networks in Unsupervised Feature Learning

  2011cited by this paper
• Melting of Peridotite to 140 Gigapascals

  2010influential reference
• Recognizing indoor scenes

  2009influential reference
• Learning Multiple Layers of Features from Tiny Images

  2009cited by this paper
• Labeled Faces in the Wild: A Database forStudying Face Recognition in Unconstrained Environments

  2008influential reference
• Automated Flower Classification over a Large Number of Classes

  2008cited by this paper
• The self-tuning neuron: synaptic scaling of excitatory synapses.

  2008cited by this paper
• Caltech-256 Object Category Dataset

  2007influential reference
• Synaptic scaling mediated by glial TNF-α

  2006cited by this paper
• A Visual Vocabulary for Flower Classification

  2006influential reference
• Homeostatic plasticity in the developing nervous system

  2004cited by this paper
• Learning Generative Visual Models from Few Training Examples: An Incremental Bayesian Approach Tested on 101 Object Categories

  2004cited by this paper
• Biologically Plausible Error-Driven Learning Using Local Activation Differences: The Generalized Recirculation Algorithm

  1996influential reference
• Neural networks

  1995cited by this paper
• A direct adaptive method for faster backpropagation learning: the RPROP algorithm

  1993influential reference
• Darpa Timit Acoustic-Phonetic Continuous Speech Corpus CD-ROM {TIMIT} | NIST

  1993cited by this paper
• A more biologically plausible learning rule for neural networks.

  1991cited by this paper
• Multilayer feedforward networks are universal approximators

  1989cited by this paper
• The recent excitement about neural networks

  1989cited by this paper
• Learning Representations by Recirculation

  1987cited by this paper
• Competitive Learning: From Interactive Activation to Adaptive Resonance

  1987cited by this paper
• Learning Process in an Asymmetric Threshold Network

  1986cited by this paper
• Information processing in dynamical systems: foundations of harmony theory

  1986cited by this paper
• Learning representations by back-propagating errors

  1986influential reference

• Hebbian Learning with Global Direction

  2026cites this paper
• Three factor delay learning rules for spiking neural networks

  2026cites this paper
• The brain as a blueprint: a survey of brain-inspired approaches to learning in artificial intelligence

  2025cites this paper
• Dopamine-driven synaptic credit assignment in neural networks

  2025cites this paper
• ‘Backpropagation and the brain’ realized in cortical error neuron microcircuits

  2025cites this paper
• Hierarchical Reasoning Models: Perspectives and Misconceptions

  2025cites this paper
• Incorporating Visual Cortical Lateral Connection Properties into CNN: Recurrent Activation and Excitatory-Inhibitory Separation

  2025influential citation
• DRAFT: Decoupling Backpropagation from Pre-trained Backbone for Efficient Transformer Fine-Tuning on Edge

  2025influential citation
• LoRA-TCN: A Pre-Trained/Fine-Tuning Learning Paradigm for Drift Adaptation

  2025cites this paper
• Challenging Backpropagation: Evidence for Target Learning in the Neocortex

  2025cites this paper
• Eigen Neural Network: Unlocking Generalizable Vision with Eigenbasis

  2025cites this paper
• Neural Architecture Search with Mixed Bio-inspired Learning Rules

  2025cites this paper
• Spiking Auto-Encoder for Static and Spatio-Temporal Neuromorphic Pattern Reconstruction

  2025cites this paper
• Topological Insulators Boost Ultralow-Power Neuromorphic Spintronics: Advancing Handwritten Digit Recognition with High SOT Efficiency.

  2025cites this paper
• One-Time Soft Alignment Enables Resilient Learning without Weight Transport

  2025cites this paper
• Frozen Backpropagation: Relaxing Weight Symmetry in Temporally-Coded Deep Spiking Neural Networks

  2025cites this paper
• Navigating beyond backpropagation: on alternative training methods for deep neural networks

  2025cites this paper
• Noise-based reward-modulated learning

  2025cites this paper
• Weight transport through spike timing for robust local gradients

  2025cites this paper
• Backpropagation-free Spiking Neural Networks with the Forward-Forward Algorithm

  2025cites this paper
• Sign-Symmetry Learning Rules are Robust Fine-Tuners

  2025cites this paper
• Training Large Neural Networks With Low-Dimensional Error Feedback

  2025cites this paper
• Towards biologically plausible DNN optimization: Replacing backpropagation and loss functions with a top-down credit assignment network

  2025cites this paper
• Online Pseudo-Zeroth-Order Training of Neuromorphic Spiking Neural Networks

  2024cites this paper
• Self-Assembly of a Biologically Plausible Learning Circuit

  2024cites this paper
• M^3ashy: Multi-Modal Material Synthesis via Hyperdiffusion

  2024cites this paper
• Protecting Feed-Forward Networks from Adversarial Attacks Using Predictive Coding

  2024cites this paper
• Oja's plasticity rule overcomes several challenges of training neural networks under biological constraints

  2024cites this paper
• Towards Biologically Plausible Computing: A Comprehensive Comparison

  2024cites this paper
• Towards Interpretable Deep Local Learning with Successive Gradient Reconciliation

  2024cites this paper
• Direct Feedback Learning With Local Alignment Support

  2024cites this paper
• Deep Learning without Weight Symmetry

  2024cites this paper
• Pretraining with Random Noise for Fast and Robust Learning without Weight Transport

  2024cites this paper
• Contribute to balance, wire in accordance: Emergence of backpropagation from a simple, bio-plausible neuroplasticity rule

  2024cites this paper
• Dictionary Contrastive Learning for Efficient Local Supervision without Auxiliary Networks

  2024cites this paper
• Lightweight Inference for Forward-Forward Algorithm

  2024cites this paper
• A Review of Neuroscience-Inspired Machine Learning

  2024cites this paper
• Prediction Accuracy and Adversarial Robustness of Error-Based Input Perturbation Learning

  2024cites this paper
• NeuroFlux: Memory-Efficient CNN Training Using Adaptive Local Learning

  2024influential citation
• Learning Using a Single Forward Pass

  2024cites this paper
• Forward Learning of Large Language Models by Consumer Devices

  2024cites this paper
• Forward Learning with Top-Down Feedback: Empirical and Analytical Characterization

  2023cites this paper
• Mathematical Formulation of Learning and Its Computational Complexity for Transformers’ Layers

  2023cites this paper
• Interpreting learning in biological neural networks as zero-order optimization method

  2023cites this paper
• Photonic online learning: a perspective

  2023cites this paper
• Zenkai - Framework For Exploring Beyond Backpropagation

  2023cites this paper
• Brain-like Flexible Visual Inference by Harnessing Feedback-Feedforward Alignment

  2023cites this paper
• NADOL: Neuromorphic Architecture for Spike-Driven Online Learning by Dendrites

  2023cites this paper
• Improving equilibrium propagation without weight symmetry through Jacobian homeostasis

  2023cites this paper
• Brain-Inspired Computational Intelligence via Predictive Coding

  2023cites this paper
• Layer-wise Feedback Propagation

  2023cites this paper
• Neuromorphic Neuromodulation: Towards the next generation of on-device AI-revolution in electroceuticals

  2023cites this paper
• Learning with Auxiliary Activation for Memory-Efficient Training

  2023cites this paper
• Backpropagation-Free Deep Learning with Recursive Local Representation Alignment

  2023cites this paper
• Can Forward Gradient Match Backpropagation?

  2023cites this paper
• Correlative Information Maximization: A Biologically Plausible Approach to Supervised Deep Neural Networks without Weight Symmetry

  2023cites this paper
• Implicit Regularization in Feedback Alignment Learning Mechanisms for Neural Networks

  2023influential citation
• Synaptic Weight Distributions Depend on the Geometry of Plasticity

  2023cites this paper
• Understanding and Improving Optimization in Predictive Coding Networks

  2023cites this paper
• Feed-Forward Optimization With Delayed Feedback for Neural Networks

  2023cites this paper
• 融合张量合成注意力的改进ResNet图像分类模型

  2023cites this paper
• Effect of Optimization Techniques on Feedback Alignment Learning of Neural Networks

  2023cites this paper
• Brain-like Flexible Visual Inference by Harnessing Feedback Feedforward Alignment

  2023cites this paper
• Deep Learning Depends on Segregated Dendritic Compartments and Spike Frequency Adaptation

  2023cites this paper
• A Computational Framework of Cortical Microcircuits Approximates Sign-concordant Random Backpropagation

  2022influential citation
• Forward Signal Propagation Learning

  2022cites this paper
• Photons guided by axons may enable backpropagation-based learning in the brain

  2022cites this paper
• Error-driven Input Modulation: Solving the Credit Assignment Problem without a Backward Pass

  2022influential citation
• Accelerating DNN Training Through Selective Localized Learning

  2022cites this paper
• Contribution of Timbre and Shimmer Features to Deepfake Speech Detection

  2022cites this paper
• Signal Propagation: The Framework for Learning and Inference in a Forward Pass

  2022cites this paper
• Brain-inspired Predictive Coding Improves the Performance of Machine Challenging Tasks

  2022cites this paper
• Meta-learning biologically plausible plasticity rules with random feedback pathways

  2022cites this paper
• Scaling Forward Gradient With Local Losses

  2022cites this paper
• The Influence of Learning Rule on Representation Dynamics in Wide Neural Networks

  2022cites this paper
• Biologically Plausible Training of Deep Neural Networks Using a Top-down Credit Assignment Network

  2022cites this paper
• Learning Event-based Spatio-Temporal Feature Descriptors via Local Synaptic Plasticity: A Biologically-Plausible Perspective of Computer Vision

  2022cites this paper
• BioLCNet: Reward-modulated Locally Connected Spiking Neural Networks

  2021cites this paper
• Training Deep Architectures Without End-to-End Backpropagation: A Brief Survey

  2021cites this paper
• Learning Without Feedback: Fixed Random Learning Signals Allow for Feedforward Training of Deep Neural Networks

  2021cites this paper
• Frozen algorithms: how the brain's wiring facilitates learning

  2021cites this paper
• Efficient Spike-Driven Learning With Dendritic Event-Based Processing

  2021cites this paper
• Efficient Training Convolutional Neural Networks on Edge Devices with Gradient-pruned Sign-symmetric Feedback Alignment

  2021cites this paper
• Temporal Coding in Spiking Neural Networks With Alpha Synaptic Function: Learning With Backpropagation

  2021cites this paper
• Bottom-Up and Top-Down Neural Processing Systems Design: Neuromorphic Intelligence as the Convergence of Natural and Artificial Intelligence

  2021cites this paper
• Credit Assignment Through Broadcasting a Global Error Vector

  2021cites this paper
• Align, then memorise: the dynamics of learning with feedback alignment

  2021cites this paper
• The backpropagation algorithm implemented on spiking neuromorphic hardware

  2021cites this paper
• Applications of the Free Energy Principle to Machine Learning and Neuroscience

  2021cites this paper
• Tourbillon: a Physically Plausible Neural Architecture

  2021cites this paper
• A Sparsity-Driven Backpropagation-Less Learning Framework Using Populations of Spiking Growth Transform Neurons

  2021cites this paper
• Benchmarking the Accuracy and Robustness of Feedback Alignment Algorithms

  2021cites this paper
• Biologically Plausible Training Mechanisms for Self-Supervised Learning in Deep Networks

  2021cites this paper
• Learning Event-based Spatio-Temporal Feature Descriptors via Local Synaptic Plasticity: A Biologically-realistic Perspective of Computer Vision

  2021cites this paper
• Information bottleneck-based Hebbian learning rule naturally ties working memory and synaptic updates

  2021cites this paper
• How to Train Your Wide Neural Network Without Backprop: An Input-Weight Alignment Perspective

  2021influential citation
• Training Deep Architectures Without End-to-End Backpropagation: A Survey on the Provably Optimal Methods

  2021influential citation
• The underlying mechanisms of alignment in error backpropagation through arbitrary weights

  2021cites this paper
• Bottom-Up and Top-Down Approaches for the Design of Neuromorphic Processing Systems: Tradeoffs and Synergies Between Natural and Artificial Intelligence

  2021cites this paper
• Deriving Differential Target Propagation from Iterating Approximate Inverses

  2020cites this paper