Learning Sparse Neural Networks through L0 Regularization

We propose a practical method for $L_0$ norm regularization for neural networks: pruning the network during training by encouraging weights to become exactly zero. Such regularization is interesting since (1) it can greatly speed up training and inference, and (2) it can improve generalization. AIC and BIC, well-known model selection criteria, are special cases of $L_0$ regularization. However, since the $L_0$ norm of weights is non-differentiable, we cannot incorporate it directly as a regularization term in the objective function. We propose a solution through the inclusion of a collection of non-negative stochastic gates, which collectively determine which weights to set to zero. We show that, somewhat surprisingly, for certain distributions over the gates, the expected $L_0$ norm of the resulting gated weights is differentiable with respect to the distribution parameters. We further propose the \emph{hard concrete} distribution for the gates, which is obtained by "stretching" a binary concrete distribution and then transforming its samples with a hard-sigmoid. The parameters of the distribution over the gates can then be jointly optimized with the original network parameters. As a result our method allows for straightforward and efficient learning of model structures with stochastic gradient descent and allows for conditional computation in a principled way. We perform various experiments to demonstrate the effectiveness of the resulting approach and regularizer.

• Publication year

  2017

• Venue

  International Conference on Learning Representations

• Publication date

  2017-12-04

• Fields of study

  Mathematics, Computer Science

• Identifiers
  arXiv 1712.01312
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Structured Bayesian Pruning via Log-Normal Multiplicative Noise

  2017influential reference
• Bayesian Compression for Deep Learning

  2017influential reference
• Soft Weight-Sharing for Neural Network Compression

  2017cited by this paper
• Variational Dropout Sparsifies Deep Neural Networks

  2017influential reference
• REBAR: Low-variance, unbiased gradient estimates for discrete latent variable models

  2017cited by this paper
• Understanding deep learning requires rethinking generalization

  2016cited by this paper
• Identity Mappings in Deep Residual Networks

  2016cited by this paper
• Training Sparse Neural Networks

  2016cited by this paper
• Learning Structured Sparsity in Deep Neural Networks

  2016cited by this paper
• Variational Inference for Monte Carlo Objectives

  2016cited by this paper
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables

  2016influential reference
• Wide Residual Networks

  2016influential reference
• Categorical Reparameterization with Gumbel-Softmax

  2016influential reference
• Discrete Variational Autoencoders

  2016cited by this paper
• A Structured Variational Auto-encoder for Learning Deep Hierarchies of Sparse Features

  2016cited by this paper
• Variational Dropout and the Local Reparameterization Trick

  2015cited by this paper
• Conditional Computation in Neural Networks for faster models

  2015cited by this paper
• Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding

  2015influential reference
• Deep Residual Learning for Image Recognition

  2015cited by this paper
• Dropout: a simple way to prevent neural networks from overfitting

  2014cited by this paper
• Neural Variational Inference and Learning in Belief Networks

  2014cited by this paper
• Efficient object localization using Convolutional Networks

  2014cited by this paper
• Adam: A Method for Stochastic Optimization

  2014cited by this paper
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models

  2014cited by this paper
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation

  2013influential reference
• Auto-Encoding Variational Bayes

  2013cited by this paper
• Learning with structured sparsity

  2009cited by this paper
• Approximating the Kullback Leibler Divergence Between Gaussian Mixture Models

  2007cited by this paper
• Variational learning for rectified factor analysis

  2007cited by this paper
• Elements of Information Theory

  2005cited by this paper
• Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning

  2004cited by this paper
• Variational algorithms for approximate Bayesian inference

  2003cited by this paper
• Gradient-based learning applied to document recognition

  1998cited by this paper
• Generative models for discovering sparse distributed representations.

  1997cited by this paper
• Regression Shrinkage and Selection via the Lasso

  1996cited by this paper
• Simple statistical gradient-following algorithms for connectionist reinforcement learning

  1992cited by this paper
• Optimal Brain Damage

  1989cited by this paper
• Bayesian Variable Selection in Linear Regression

  1988cited by this paper
• Estimating the Dimension of a Model

  1978cited by this paper
• Information Theory and an Extension of the Maximum Likelihood Principle

  1973cited by this paper

• Entropy-Guided Condensing for Vision Transformer

  2026cites this paper
• Rational ANOVA Networks

  2026cites this paper
• Weights to Code: Extracting Interpretable Algorithms from the Discrete Transformer

  2026cites this paper
• Variational Sparse Paired Autoencoders (vsPAIR) for Inverse Problems and Uncertainty Quantification

  2026cites this paper
• Concept-Aware Privacy Mechanisms for Defending Embedding Inversion Attacks

  2026cites this paper
• Pruning as Evolution: Emergent Sparsity Through Selection Dynamics in Neural Networks

  2026cites this paper
• LUMOS: Democratizing SciML Workflows with L0-Regularized Learning for Unified Feature and Parameter Adaptation

  2026cites this paper
• Enhancing LUT-based Deep Neural Networks Inference through Architecture and Connectivity Optimization

  2026cites this paper
• Revisiting Adaptive Rounding with Vectorized Reparameterization for LLM Quantization

  2026cites this paper
• QExpander: Pruning-sensitivity-guided adaptive widening for binary neural networks

  2026cites this paper
• Building Envelope Inversion by Data-driven Interpretation of Ground Penetrating Radar

  2026cites this paper
• Sparse Gaussian–Markov Modeling for Robust and Trustworthy Unknown Cyber Defense

  2026influential citation
• Clarity: The Flexibility-Interpretability Trade-Off in Sparsity-aware Concept Bottleneck Models

  2026influential citation
• T3C: Test-Time Tensor Compression with Consistency Guarantees

  2026cites this paper
• ECSEL: Explainable Classification via Signomial Equation Learning

  2026cites this paper
• SparseX: Synergizing GPU Libraries for Sparse Matrix Multiplication on Heterogeneous Processors

  2026cites this paper
• Effective Model Pruning

  2025cites this paper
• $\ell_0$-Norm Multiobjective Optimization Models Motivated by Applications to Proton Therapy

  2025cites this paper
• Unsupervised Dynamic Feature Selection for Robust Latent Spaces in Vision Tasks

  2025influential citation
• Reinforcement learning in densely recurrent biological networks

  2025cites this paper
• Masks Can be Learned as an Alternative to Experts

  2025cites this paper
• Crafting Imperceptible On-Manifold Adversarial Attacks for Tabular Data

  2025cites this paper
• Semantic-aware DropSplat: Adaptive Pruning of Redundant Gaussians for 3D Aerial-View Segmentation

  2025cites this paper
• Physics Augmented Machine Learning Discovery of Composition-Dependent Constitutive Laws for 3D Printed Digital Materials

  2025influential citation
• WEEP: A Differentiable Nonconvex Sparse Regularizer via Weakly-Convex Envelope

  2025cites this paper
• Efficient compression of wav2vec 2.0 for edge deployment in speech emotion & speaker recognition

  2025cites this paper
• Real-time lightweight strawberry ripeness detection framework based on YOLO11 deployed on edge computing platform

  2025cites this paper
• K-sparse encoder for efficient information retrieval

  2025cites this paper
• Probabilistic Kolmogorov-Arnold Networks via Sparsified Deep Gaussian Processes with Additive Kernels

  2025cites this paper
• (Sometimes) Less is More: Mitigating the Complexity of Rule-based Representation for Interpretable Classification

  2025influential citation
• StructPrune: Structured Global Pruning asymptotics with 𝒪(√N) GPU Memory

  2025cites this paper
• Automatic Expert Discovery in LLM Upcycling via Sparse Interpolated Mixture-of-Experts

  2025influential citation
• Global Minimizers of ℓp-Regularized Objectives Yield the Sparsest ReLU Neural Networks

  2025cites this paper
• A Novel Compiler Transformation for Fast Sparse Matrix Multiplication in GPUs

  2025cites this paper
• Super-fast rates of convergence for Neural Networks Classifiers under the Hard Margin Condition

  2025cites this paper
• AdaRank: Adaptive Rank Pruning for Enhanced Model Merging

  2025cites this paper
• The Neural Pruning Law Hypothesis

  2025cites this paper
• Text-to-Voice Conversion for Indic Languages

  2025cites this paper
• NeuronSeek: On Stability and Expressivity of Task-driven Neurons

  2025cites this paper
• V-CEM: Bridging Performance and Intervenability in Concept-based Models

  2025cites this paper
• Context-aware Dynamic Pruning for Speech Foundation Models

  2025influential citation
• WAPS-Quant: Low-Bit Post-Training Quantization Using Weight-Activation Product Scaling

  2025cites this paper
• Robust extrapolation using physics-related activation functions in neural networks for nuclear masses

  2025influential citation
• Position: Adopt Constraints Over Penalties in Deep Learning

  2025influential citation
• Inference-Time Hyper-Scaling with KV Cache Compression

  2025cites this paper
• Machine Unlearning for Robust DNNs: Attribution-Guided Partitioning and Neuron Pruning in Noisy Environments

  2025cites this paper
• Towards Interpretable Adversarial Examples via Sparse Adversarial Attack

  2025cites this paper
• Learning Interpretable Rules from Neural Networks: Neurosymbolic AI for Radar Hand Gesture Recognition

  2025cites this paper
• Generative Models Reimagined: Review into Resource-Efficient Training

  2025cites this paper
• Know Your Attention Maps: Class-specific Token Masking for Weakly Supervised Semantic Segmentation

  2025cites this paper
• Learning from Scratch: Structurally-masked Transformer for Next Generation Lib-free Simulation

  2025cites this paper
• Efficient and Generalizable Speaker Diarization via Structured Pruning of Self-Supervised Models

  2025influential citation
• Sparsity-Driven Plasticity in Multi-Task Reinforcement Learning

  2025cites this paper
• FAIR-Pruner: Leveraging Tolerance of Difference for Flexible Automatic Layer-Wise Neural Network Pruning

  2025cites this paper
• Hybrid Pruning: In-Situ Compression of Self-Supervised Speech Models for Speaker Verification and Anti-Spoofing

  2025cites this paper
• Z -Prunear: Post-Training Pruning of Large Language Models for Efficiency without Retraining

  2025cites this paper
• LSA-MEP: layer-wise sparsity allocation multi-metric evaluation pruning

  2025cites this paper
• VISP: Volatility Informed Stochastic Projection for Adaptive Regularization

  2025cites this paper
• Global Interpretability for ProtoPNet Using Rule-Based Explanations

  2025cites this paper
• Probabilistic and nonlinear compressive sensing

  2025cites this paper
• WideTopo: Improving foresight neural network pruning through training dynamics preservation and wide topologies exploration

  2025cites this paper
• Exact Subgraph Isomorphism Network with Mixed $L_{0,2}$ Norm Constraint for Predictive Graph Mining

  2025cites this paper
• SparseLUT: Sparse Connectivity Optimization for Lookup Table-Based Deep Neural Networks

  2025cites this paper
• A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applications

  2025cites this paper
• FP4DiT: Towards Effective Floating Point Quantization for Diffusion Transformers

  2025influential citation
• Climate And Resource Awareness is Imperative to Achieving Sustainable AI (and Preventing a Global AI Arms Race)

  2025cites this paper
• prunAdag: an adaptive pruning-aware gradient method

  2025cites this paper
• MaskPrune: Mask-based LLM Pruning for Layer-wise Uniform Structures

  2025cites this paper
• Input Specific Neural Networks

  2025cites this paper
• Information Theoretic Pruning of Coupled Channels in Deep Neural Networks

  2025cites this paper
• Sparse Data Diffusion for Scientific Simulations in Biology and Physics

  2025cites this paper
• Overcoming sparse datasets with multi-task learning as applied to high entropy alloys

  2025cites this paper
• On-device edge learning for IoT data streams: a survey

  2025cites this paper
• DRAMA: Diverse Augmentation from Large Language Models to Smaller Dense Retrievers

  2025cites this paper
• Polyconvex Physics-Augmented Neural Network Constitutive Models in Principal Stretches

  2025cites this paper
• A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Models

  2025cites this paper
• IDEA Prune: An Integrated Enlarge-and-Prune Pipeline in Generative Language Model Pretraining

  2025cites this paper
• SSVQ: Unleashing the Potential of Vector Quantization with Sign-Splitting

  2025cites this paper
• Lipschitz Constant Meets Condition Number: Learning Robust and Compact Deep Neural Networks

  2025cites this paper
• Stochastic Engrams for Efficient Continual Learning with Binarized Neural Networks

  2025cites this paper
• An Efficient Training Algorithm for Models with Block-wise Sparsity

  2025cites this paper
• SparsyFed: Sparse Adaptive Federated Training

  2025cites this paper
• HEK-CL: Hierarchical Enhanced Knowledge-Aware Contrastive Learning for Recommendation

  2025cites this paper
• DyExplainer: Self-explainable Dynamic Graph Neural Network with Sparse Attentions

  2025cites this paper
• Improving interpretability for fault diagnosis in complex chemical processes using combined regularization and linearization in neural network modeling

  2025cites this paper
• A Lightweight Cloud-Edge Collaborative Intelligence Inference Framework With Runtime Dynamic Optimization for Resource-Constrained Consumer Electronics

  2025cites this paper
• DPaI: Differentiable Pruning at Initialization with Node-Path Balance Principle

  2025cites this paper
• MID-L: Matrix-Interpolated Dropout Layer with Layer-wise Neuron Selection

  2025cites this paper
• TRIM: Achieving Extreme Sparsity with Targeted Row-wise Iterative Metric-driven Pruning

  2025cites this paper
• Context-Driven Dynamic Pruning for Large Speech Foundation Models

  2025cites this paper
• DenoiseRotator: Enhance Pruning Robustness for LLMs via Importance Concentration

  2025cites this paper
• Fine-tune Before Structured Pruning: Towards Compact and Accurate Self-Supervised Models for Speaker Diarization

  2025influential citation
• Line of Sight: On Linear Representations in VLLMs

  2025cites this paper
• Hyperpruning: Efficient Search through Pruned Variants of Recurrent Neural Networks Leveraging Lyapunov Spectrum

  2025cites this paper
• Zero-Aware Regularization for Energy-Efficient Inference on Akida Neuromorphic Processor

  2025cites this paper
• Semantic ECG hash similarity graph

  2025cites this paper
• TopK Language Models

  2025cites this paper
• Projected Compression: Trainable Projection for Efficient Transformer Compression

  2025cites this paper
• Minimalist Concept Erasure in Generative Models

  2025cites this paper
• Deep Weight Factorization: Sparse Learning Through the Lens of Artificial Symmetries

  2025cites this paper