A Hebbian/Anti-Hebbian network derived from online non-negative matrix factorization can cluster and discover sparse features

Despite our extensive knowledge of biophysical properties of neurons, there is no commonly accepted algorithmic theory of neuronal function. Here we explore the hypothesis that single-layer neuronal networks perform online symmetric nonnegative matrix factorization (SNMF) of the similarity matrix of the streamed data. By starting with the SNMF cost function we derive an online algorithm, which can be implemented by a biologically plausible network with local learning rules. We demonstrate that such network performs soft clustering of the data as well as sparse feature discovery. The derived algorithm replicates many known aspects of sensory anatomy and biophysical properties of neurons including unipolar nature of neuronal activity and synaptic weights, local synaptic plasticity rules and the dependence of learning rate on cumulative neuronal activity. Thus, we make a step towards an algorithmic theory of neuronal function, which should facilitate large-scale neural circuit simulations and biologically inspired artificial intelligence.

• Publication year

  2014

• Venue

  Asilomar Conference on Signals, Systems and Computers

• Publication date

  2014-11-01

• Fields of study

  Biology, Mathematics, Computer Science

• Identifiers
  DOI 10.1109/ACSSC.2014.7094553arXiv 1503.00680
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• A Hebbian/Anti-Hebbian Neural Network for Linear Subspace Learning: A Derivation from Multidimensional Scaling of Streaming Data

  2015cited by this paper
• K-means recovers ICA filters when independent components are sparse

  2014influential reference
• A Hebbian/Anti-Hebbian network for online sparse dictionary learning derived from symmetric matrix factorization

  2014cited by this paper
• A neuron as a signal processing device

  2013cited by this paper
• Symmetric Nonnegative Matrix Factorization for Graph Clustering

  2012influential reference
• Learning Feature Representations with K-Means

  2012influential reference
• An Analysis of Single-Layer Networks in Unsupervised Feature Learning

  2011cited by this paper
• Normalization as a canonical neural computation

  2011cited by this paper
• Clustering: A neural network approach

  2010cited by this paper
• An Early Critical Period for Long-Term Plasticity and Structural Modification of Sensory Synapses in Olfactory Cortex

  2007cited by this paper
• On the Equivalence of Nonnegative Matrix Factorization and Spectral Clustering

  2005cited by this paper
• Self-organized formation of topologically correct feature maps

  2004cited by this paper
• Sparse coding of sensory inputs.

  2004cited by this paper
• Independent component analysis: algorithms and applications

  2000influential reference
• The "independent components" of natural scenes are edge filters.

  1997cited by this paper
• The nonlinear PCA learning rule in independent component analysis

  1997cited by this paper
• Sparse coding with an overcomplete basis set: a strategy employed by V1?

  1997cited by this paper
• Emergence of simple-cell receptive field properties by learning a sparse code for natural images

  1996cited by this paper
• A critical period for long-term potentiation at thalamocortical synapses

  1995cited by this paper
• Co-regulation of long-term potentiation and experience-dependent synaptic plasticity in visual cortex by age and experience

  1995cited by this paper
• Introduction to the theory of neural computation

  1994cited by this paper
• Principal components, minor components, and linear neural networks

  1992cited by this paper
• What Does the Retina Know about Natural Scenes?

  1992cited by this paper
• A neural network learning algorithm for adaptive principal component extraction (APEX)

  1990cited by this paper
• Optimal unsupervised learning in a single-layer linear feedforward neural network

  1989cited by this paper
• Adaptive network for optimal linear feature extraction

  1989cited by this paper
• Kohonen's self-organizing maps: exploring their computational capabilities

  1988cited by this paper
• A massively parallel architecture for a self-organizing neural pattern recognition machine

  1988cited by this paper
• ART 2: self-organization of stable category recognition codes for analog input patterns.

  1987cited by this paper
• Simplified neuron model as a principal component analyzer

  1982cited by this paper
• Some methods for classification and analysis of multivariate observations

  1967cited by this paper
• Receptive fields, binocular interaction and functional architecture in the cat's visual cortex

  1962cited by this paper

• Policy optimization emerges from noisy representation learning

  2025cites this paper
• Similarity Matching Networks: Hebbian Learning and Convergence Over Multiple Time Scales

  2025cites this paper
• Implicit Generative Modeling by Kernel Similarity Matching

  2025cites this paper
• Normative Framework for Deriving Neural Networks with Multicompartmental Neurons and Non-Hebbian Plasticity

  2023cites this paper
• Coordinated drift of receptive fields in Hebbian/anti-Hebbian network models during noisy representation learning

  2023cites this paper
• A normative framework for deriving neural networks with multi-compartmental neurons and non-Hebbian plasticity

  2023cites this paper
• Duality Principle and Biologically Plausible Learning: Connecting the Representer Theorem and Hebbian Learning

  2023cites this paper
• Kernel similarity matching with Hebbian neural networks

  2022cites this paper
• Kernel similarity matching with Hebbian networks

  2022cites this paper
• Biologically-Plausible Determinant Maximization Neural Networks for Blind Separation of Correlated Sources

  2022cites this paper
• Revisiting Gaussian Neurons for Online Clustering with Unknown Number of Clusters

  2022cites this paper
• A Normative and Biologically Plausible Algorithm for Independent Component Analysis

  2021cites this paper
• A Similarity-preserving Neural Network Trained on Transformed Images Recapitulates Salient Features of the Fly Motion Detection Circuit

  2021influential citation
• Normative and mechanistic model of an adaptive circuit for efficient encoding and feature extraction

  2021cites this paper
• Coordinated drift of receptive fields during noisy representation learning

  2021cites this paper
• SoftHebb: Bayesian inference in unsupervised Hebbian soft winner-take-all networks

  2021cites this paper
• Neural circuits for dynamics-based segmentation of time series

  2021cites this paper
• Blind Bounded Source Separation Using Neural Networks with Local Learning Rules

  2020cites this paper
• Network Dynamics Governed by Lyapunov Functions: From Memory to Classification.

  2020cites this paper
• Directional Sparse Filtering for Blind Estimation of Under-Determined Complex-Valued Mixing Matrices

  2020cites this paper
• Structured and Deep Similarity Matching via Structured and Deep Hebbian Networks

  2019cites this paper
• Training Neural Networks with Local Error Signals

  2019cites this paper
• A Spiking Neural Network with Local Learning Rules Derived from Nonnegative Similarity Matching

  2019cites this paper
• Biologically plausible deep learning - but how far can we go with shallow networks?

  2019cites this paper
• Neuroscience-Inspired Online Unsupervised Learning Algorithms: Artificial neural networks

  2019cites this paper
• Unsupervised learning by a "softened" correlation game: duality and convergence

  2019influential citation
• Unsupervised learning by a nonlinear network with Hebbian excitatory and anti-Hebbian inhibitory neurons

  2018cites this paper
• Odor Perception on the Two Sides of the Brain: Consistency Despite Randomness.

  2018cites this paper
• Manifold-tiling Localized Receptive Fields are Optimal in Similarity-preserving Neural Networks

  2018influential citation
• The Neuronal Basis of an Illusory Motion Percept Is Explained by Decorrelation of Parallel Motion Pathways.

  2018cites this paper
• Unsupervised learning by competing hidden units

  2018cites this paper
• Neural Networks for Efficient Nonlinear Online Clustering

  2017influential citation
• Building Efficient Deep Hebbian Networks for Image Classification Tasks

  2017cites this paper
• Blind Nonnegative Source Separation Using Biological Neural Networks

  2017influential citation
• Hebbian plasticity requires compensatory processes on multiple timescales

  2017cites this paper
• Online Representation Learning with Multi-layer Hebbian Networks for Image Classification Tasks

  2017influential citation
• A clustering neural network model of insect olfaction

  2017influential citation
• A correlation game for unsupervised learning yields computational interpretations of Hebbian excitation, anti-Hebbian inhibition, and synapse elimination

  2017cites this paper
• Why Do Similarity Matching Objectives Lead to Hebbian/Anti-Hebbian Networks?

  2017cites this paper
• Adversarial synapses: Hebbian/anti-Hebbian learning optimizes min-max objectives

  2017cites this paper
• Online Representation Learning with Single and Multi-layer Hebbian Networks for Image Classification

  2017influential citation
• Self-calibrating neural networks for dimensionality reduction

  2016cites this paper
• Recovery Guarantee of Non-negative Matrix Factorization via Alternating Updates

  2016cites this paper
• Optimization theory of Hebbian/anti-Hebbian networks for PCA and whitening

  2015cites this paper
• A Hebbian/Anti-Hebbian network for online sparse dictionary learning derived from symmetric matrix factorization

  2014cites this paper