Delving into Unsupervised Hebbian Learning from Artificial Intelligence Perspectives

Unsupervised Hebbian learning is a biologically inspired algorithm designed to extract representations from input images, which can subsequently support supervised learning. It presents a promising alternative to traditional artificial neural networks (ANNs). Many attempts have focused on enhancing Hebbian learning by incorporating more biologically plausible components. Contrarily, we draw inspiration from recent advances in ANNs to rethink and further improve Hebbian learning in three interconnected aspects. First, we investigate the issue of overfitting in Hebbian learning and emphasize the importance of selecting an optimal number of training epochs, even in unsupervised settings. In addition, we discuss the risks and benefits of anti-Hebbian learning in model performance, and our visualizations reveal that synapses resembling the input images sometimes do not necessarily reflect effective learning. Then, we explore the impact of different activation functions on Hebbian representations, highlighting the benefits of properly utilizing negative values. Furthermore, motivated by the success of large pre-trained language models, we propose a novel approach for leveraging unlabeled data from other datasets. Unlike conventional pre-training in ANNs, experimental results demonstrate that merging trained synapses from different datasets leads to improved performance. Overall, our findings offer fresh perspectives on enhancing the future design of Hebbian learning algorithms.

• Publication year

  2025

• Venue

  Machine Learning and Knowledge Extraction

• Publication date

  2025-11-11

• Fields of study

  Computer Science

• Identifiers
  DOI 10.3390/make7040143
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Rethinking the Masking Strategy for Pretraining Molecular Graphs from a Data-Centric View

  2024influential reference
• Intelligent cell images segmentation system: based on SDN and moving transformer

  2024cited by this paper
• Development of Bio‐Voltage Operated Humidity‐Sensory Neurons Comprising Self‐Assembled Peptide Memristors

  2024cited by this paper
• Utilizing data imbalance to enhance compound-protein interaction prediction models

  2024cited by this paper
• Scalable bio-inspired training of Deep Neural Networks with FastHebb

  2024cited by this paper
• NEFTune: Noisy Embeddings Improve Instruction Finetuning

  2023cited by this paper
• Brain-inspired learning in artificial neural networks: a review

  2023cited by this paper
• Competition on presynaptic resources enhances the discrimination of interfering memories

  2023cited by this paper
• GPT-4 Technical Report

  2023influential reference
• Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time

  2022cited by this paper
• A ConvNet for the 2020s

  2022cited by this paper
• Hebbian Deep Learning Without Feedback

  2022cited by this paper
• HebbNet: A Simplified Hebbian Learning Framework to do Biologically Plausible Learning

  2021cited by this paper
• BatchDTA: Implicit batch alignment enhances deep learning-based drug-target affinity estimation

  2021influential reference
• Masked Autoencoders Are Scalable Vision Learners

  2021cited by this paper
• SoftHebb: Bayesian inference in unsupervised Hebbian soft winner-take-all networks

  2021cited by this paper
• BEiT: BERT Pre-Training of Image Transformers

  2021cited by this paper
• ProtTrans: Towards Cracking the Language of Life’s Code Through Self-Supervised Deep Learning and High Performance Computing

  2020cited by this paper
• The unreasonable effectiveness of deep learning in artificial intelligence

  2020cited by this paper
• 5分で分かる!? 有名論文ナナメ読み：Jacob Devlin et al. : BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding

  2020influential reference
• The overfitted brain: Dreams evolved to assist generalization

  2020cited by this paper
• GLU Variants Improve Transformer

  2020cited by this paper
• Overfitting Mechanism and Avoidance in Deep Neural Networks

  2019cited by this paper
• Biologically plausible deep learning - but how far can we go with shallow networks?

  2019cited by this paper
• Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences

  2019influential reference
• Strategies for Pre-training Graph Neural Networks

  2019cited by this paper
• Searching for Activation Functions

  2018cited by this paper
• Deep Learning for Classical Japanese Literature

  2018cited by this paper
• Unsupervised learning by competing hidden units

  2018influential reference
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms

  2017cited by this paper
• Dense Associative Memory for Pattern Recognition

  2016cited by this paper
• Deep Learning

  2016cited by this paper
• Gaussian Error Linear Units (GELUs)

  2016cited by this paper
• Language Modeling with Gated Convolutional Networks

  2016cited by this paper
• Deep Residual Learning for Image Recognition

  2015cited by this paper
• Towards Biologically Plausible Deep Learning

  2015cited by this paper
• A Hebbian/Anti-Hebbian Neural Network for Linear Subspace Learning: A Derivation from Multidimensional Scaling of Streaming Data

  2015cited by this paper
• Rethinking the Inception Architecture for Computer Vision

  2015cited by this paper
• Dropout: a simple way to prevent neural networks from overfitting

  2014cited by this paper
• Rectifier Nonlinearities Improve Neural Network Acoustic Models

  2013cited by this paper
• Visualizing and Understanding Convolutional Networks

  2013cited by this paper
• Adaptive dropout for training deep neural networks

  2013cited by this paper
• Deep Sparse Rectifier Neural Networks

  2011cited by this paper
• Rectified Linear Units Improve Restricted Boltzmann Machines

  2010cited by this paper
• On the Computational Power of Winner-Take-All

  2000cited by this paper
• An Embarrassment of Riches

  1992cited by this paper
• Forming sparse representations by local anti-Hebbian learning

  1990cited by this paper
• The organization of behavior

  1988cited by this paper
• The organization of behavior: A neuropsychological theory

  1951cited by this paper

• No citing papers are available for this paper.