Local Learning in RRAM Neural Networks with Sparse Direct Feedback Alignment

Neural networks utilizing non-volatile random access memory (NVM) exhibit excellent power reduction over traditional CMOS implementations. RRAM (resistive random access memory) is one such emerging memory technology offering low energy, good endurance, and a large analog conductance window. When implemented in a crossbar architecture, these networks are able to bypass the von-Neumann bottleneck by performing compute in-memory. This architecture works well for inference; however, training the network is far more challenging. Networks built using RRAM can be trained on-chip with gradient descent or off-chip where weights are transferred. Backpropagation, while effective in training von-Neumann architectures, is inefficient when memory and compute are partitioned together. Commonly referred to as the weight transport problem, each neuron's dependence on the weights and errors located deeper in the network requires reading the weights in each layer before computing and applying the error. This presents a key challenge in performing efficient on chip training for non von-Neumann architectures. In this work we demonstrate an alternative to backpropagation called sparse direct feedback alignment which bypasses the weight transport problem. We simulate crossbars of HfOx RRAM based on experimental data to explore the performance, area, and energy trade-offs of using bio-plausible algorithms on the MNIST and EMNIST datasets.

• Publication year

  2019

• Venue

  International Symposium on Low Power Electronics and Design

• Publication date

  2019-07-01

• Fields of study

  Computer Science, Engineering

• Identifiers
  DOI 10.1109/ISLPED.2019.8824820
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Direct Feedback Alignment With Sparse Connections for Local Learning

  2019influential reference
• Efficient and self-adaptive in-situ learning in multilayer memristor neural networks

  2018cited by this paper
• Training Deep Convolutional Neural Networks with Resistive Cross-Point Devices

  2017cited by this paper
• Reservoir computing using dynamic memristors for temporal information processing

  2017cited by this paper
• The Role of Ti Buffer Layer Thickness on the Resistive Switching Properties of Hafnium Oxide-Based Resistive Switching Memories.

  2017cited by this paper
• EMNIST: an extension of MNIST to handwritten letters

  2017cited by this paper
• Random synaptic feedback weights support error backpropagation for deep learning

  2016cited by this paper
• Direct Feedback Alignment Provides Learning in Deep Neural Networks

  2016cited by this paper
• A Compact Model for Metal–Oxide Resistive Random Access Memory With Experiment Verification

  2016cited by this paper
• Memristor-Based Multilayer Neural Networks With Online Gradient Descent Training

  2015cited by this paper
• Enabling back propagation training of memristor crossbar neuromorphic processors

  2014cited by this paper
• A million spiking-neuron integrated circuit with a scalable communication network and interface

  2014cited by this paper
• New Generation of Predictive Technology Model for Sub-45 nm Early Design Exploration

  2006cited by this paper
• Handwritten Digit Recognition with a Back-Propagation Network

  1989cited by this paper

• Continual learning with a predictive coding based classifier

  2026cites this paper
• Frozen Backpropagation: Relaxing Weight Symmetry in Temporally-Coded Deep Spiking Neural Networks

  2025cites this paper
• When In-memory Computing Meets Spiking Neural Networks - A Perspective on Device-Circuit-System-and-Algorithm Co-design

  2024cites this paper
• Beyond Backpropagation: Optimization with Multi-Tangent Forward Gradients

  2024cites this paper
• Feed-Forward Optimization With Delayed Feedback for Neural Networks

  2023cites this paper
• Deep Predictive Coding with Bi-directional Propagation for Classification and Reconstruction

  2023cites this paper
• Feed-Forward Optimization With Delayed Feedback for Neural Network Training

  2023cites this paper
• Hebbian Deep Learning Without Feedback

  2022influential citation
• Merged Logic and Memory Fabrics for Accelerating Machine Learning Workloads

  2021cites this paper
• Hardware-Based Spiking Neural Network Using a TFT-Type AND Flash Memory Array Architecture Based on Direct Feedback Alignment

  2021cites this paper
• Learning Without Feedback: Fixed Random Learning Signals Allow for Feedforward Training of Deep Neural Networks

  2021cites this paper
• A Sparsity-Driven Backpropagation-Less Learning Framework Using Populations of Spiking Growth Transform Neurons

  2021cites this paper
• Light-in-the-loop: using a photonics co-processor for scalable training of neural networks

  2020cites this paper
• The neural coding framework for learning generative models

  2020cites this paper