k-Sparse Autoencoders

Recently, it has been observed that when representations are learnt in a way that encourages sparsity, improved performance is obtained on classification tasks. These methods involve combinations of activation functions, sampling steps and different kinds of penalties. To investigate the effectiveness of sparsity by itself, we propose the k-sparse autoencoder, which is an autoencoder with linear activation function, where in hidden layers only the k highest activities are kept. When applied to the MNIST and NORB datasets, we find that this method achieves better classification results than denoising autoencoders, networks trained with dropout, and RBMs. k-sparse autoencoders are simple to train and the encoding stage is very fast, making them well-suited to large problem sizes, where conventional sparse coding algorithms cannot be applied.

• Publication year

  2013

• Venue

  International Conference on Learning Representations

• Publication date

  2013-12-19

• Fields of study

  Mathematics, Computer Science

• Identifiers
  arXiv 1312.5663
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Improving neural networks by preventing co-adaptation of feature detectors

  2012cited by this paper
• The Importance of Encoding Versus Training with Sparse Coding and Vector Quantization

  2011cited by this paper
• An Analysis of Single-Layer Networks in Unsupervised Feature Learning

  2011cited by this paper
• Learning Fast Approximations of Sparse Coding

  2010cited by this paper
• Fast Inference in Sparse Coding Algorithms with Applications to Object Recognition

  2010cited by this paper
• Efficient Learning of Deep Boltzmann Machines

  2010cited by this paper
• Why Does Unsupervised Pre-training Help Deep Learning?

  2010cited by this paper
• 3D Object Recognition with Deep Belief Nets

  2009influential reference
• Coherence analysis of iterative thresholding algorithms

  2009cited by this paper
• Extracting and composing robust features with denoising autoencoders

  2008cited by this paper
• Iterative Hard Thresholding for Compressed Sensing

  2008cited by this paper
• Kernel Codebooks for Scene Categorization

  2008cited by this paper
• Sparse deep belief net model for visual area V2

  2007cited by this paper
• Signal Recovery From Random Measurements Via Orthogonal Matching Pursuit

  2007cited by this paper
• Supporting Online Material for Reducing the Dimensionality of Data with Neural Networks

  2006cited by this paper
• Greedy Layer-Wise Training of Deep Networks

  2006cited by this paper
• K-SVD : DESIGN OF DICTIONARIES FOR SPARSE REPRESENTATION

  2005cited by this paper
• Learning methods for generic object recognition with invariance to pose and lighting

  2004cited by this paper
• Optimally sparse representation in general (nonorthogonal) dictionaries via ℓ1 minimization

  2003cited by this paper
• Method of optimal directions for frame design

  1999cited by this paper
• Sparse coding with an overcomplete basis set: a strategy employed by V1?

  1997cited by this paper

• Towards Interpretable Hallucination Analysis and Mitigation in LVLMs via Contrastive Neuron Steering

  2026cites this paper
• TimeSAE: Sparse Decoding for Faithful Explanations of Black-Box Time Series Models

  2026cites this paper
• LANGSAE EDITING: Improving Multilingual Information Retrieval via Post-hoc Language Identity Removal

  2026cites this paper
• Cross-Modal Redundancy and the Geometry of Vision-Language Embeddings

  2026cites this paper
• Identifying Intervenable and Interpretable Features via Orthogonality Regularization

  2026cites this paper
• MonoLoss: A Training Objective for Interpretable Monosemantic Representations

  2026influential citation
• Sparse autoencoders reveal organized biological knowledge but minimal regulatory logic in single-cell foundation models: a comparative atlas of Geneformer and scGPT

  2026cites this paper
• From Knots to Knobs: Towards Steerable Collaborative Filtering Using Sparse Autoencoders

  2026influential citation
• AR&D: A Framework for Retrieving and Describing Concepts for Interpreting AudioLLMs

  2026cites this paper
• Causal Circuit Tracing Reveals Distinct Computational Architectures in Single-Cell Foundation Models: Inhibitory Dominance, Biological Coherence, and Cross-Model Convergence

  2026cites this paper
• Rethinking Representativeness and Diversity in Dynamic Data Selection

  2026cites this paper
• Joint Training Across Multiple Activation Sparsity Regimes

  2026cites this paper
• Interpreting and Controlling LLM Reasoning through Integrated Policy Gradient

  2026influential citation
• Revisiting Modality Invariance in a Multilingual Speech-Text Model via Neuron-Level Analysis

  2026cites this paper
• Protein Circuit Tracing via Cross-layer Transcoders

  2026cites this paper
• GOLF: A Generative AI Framework for Pathogenicity Prediction of Myocilin OLF Variants

  2025cites this paper
• A Non-Parametric Bayesian Approach Towards OnlineSequence Learning

  2025cites this paper
• Sparse Autoencoders Find Partially Interpretable Features in Italian Small Language Models

  2025influential citation
• A Geometric Unification of Concept Learning with Concept Cones

  2025cites this paper
• Superposition disentanglement of neural representations reveals hidden alignment

  2025cites this paper
• MicroEdit: Neuron-level Knowledge Disentanglement and Localization in Lifelong Model Editing

  2025cites this paper
• Interpretable and Steerable Concept Bottleneck Sparse Autoencoders

  2025cites this paper
• Selective Concept Bottleneck Models Without Predefined Concepts

  2025influential citation
• Sparse Autoencoders in Collaborative Filtering Enhanced LLM-based Recommender Systems

  2025cites this paper
• Air Pollution Forecasting Using Autoencoders: A Classification-Based Prediction of NO2, PM10, and SO2 Concentrations

  2025cites this paper
• A Unified Theory of Sparse Dictionary Learning in Mechanistic Interpretability: Piecewise Biconvexity and Spurious Minima

  2025cites this paper
• Use Sparse Autoencoders to Discover Unknown Concepts, Not to Act on Known Concepts

  2025cites this paper
• Generative Giants, Retrieval Weaklings: Why do Multimodal Large Language Models Fail at Multimodal Retrieval?

  2025cites this paper
• Beyond Black Boxes: Enhancing Interpretability of Transformers Trained on Neural Data

  2025cites this paper
• On the Limits of Sparse Autoencoders: A Theoretical Framework and Reweighted Remedy

  2025cites this paper
• Towards Atoms of Large Language Models

  2025cites this paper
• Mechanistic Interpretability with Sparse Autoencoder Neural Operators

  2025influential citation
• Specialization after Generalization: Towards Understanding Test-Time Training in Foundation Models

  2025influential citation
• EB-gMCR: Energy-Based Generative Modeling for Signal Unmixing and Multivariate Curve Resolution

  2025cites this paper
• CytoSAE: Interpretable Cell Embeddings for Hematology

  2025cites this paper
• A Single Direction of Truth: An Observer Model's Linear Residual Probe Exposes and Steers Contextual Hallucinations

  2025cites this paper
• Towards Interpretable and Inference-Optimal COT Reasoning with Sparse Autoencoder-Guided Generation

  2025cites this paper
• Projecting Assumptions: The Duality Between Sparse Autoencoders and Concept Geometry

  2025cites this paper
• Iterative Exploration-Driven Sparse SDP Clustering via Thompson Sampling

  2025cites this paper
• From Flat to Hierarchical: Extracting Sparse Representations with Matching Pursuit

  2025cites this paper
• Pulling Back the Curtain on Deep Networks

  2025cites this paper
• Semantic Convergence: Investigating Shared Representations Across Scaled LLMs

  2025cites this paper
• Representational Difference Explanations

  2025cites this paper
• Self-supervised large-scale kidney abnormality detection in drug safety assessment studies

  2025cites this paper
• Crosscoding Through Time: Tracking Emergence & Consolidation Of Linguistic Representations Throughout LLM Pretraining

  2025cites this paper
• SAGEA: Sparse Autoencoder-based Group Embeddings Aggregation for Fairness-Preserving Group Recommendations

  2025influential citation
• Sparse Autoencoders, Again?

  2025influential citation
• Explaining How Visual, Textual and Multimodal Encoders Share Concepts

  2025cites this paper
• A gradient of complementary learning systems emerges through meta-learning

  2025cites this paper
• Re-envisioning Euclid Galaxy Morphology: Identifying and Interpreting Features with Sparse Autoencoders

  2025cites this paper
• Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safety

  2025cites this paper
• Visual Exploration of Feature Relationships in Sparse Autoencoders with Curated Concepts

  2025cites this paper
• FoldSAE: Learning to Steer Protein Folding Through Sparse Representations

  2025cites this paper
• Consecutive Frame Extrapolation with Predictive Sparse Shading

  2025cites this paper
• SAeUron: Interpretable Concept Unlearning in Diffusion Models with Sparse Autoencoders

  2025cites this paper
• Rethinking Explainability in the Era of Multimodal AI

  2025cites this paper
• Patch-level phenotype identification via weakly supervised neuron selection in sparse autoencoders for CLIP-derived pathology embeddings.

  2025cites this paper
• Secret mixtures of experts inside your LLM

  2025cites this paper
• Inference-Time Decomposition of Activations (ITDA): A Scalable Approach to Interpreting Large Language Models

  2025cites this paper
• Enhancing Automated Interpretability with Output-Centric Feature Descriptions

  2025cites this paper
• ConSim: Measuring Concept-Based Explanations' Effectiveness with Automated Simulatability

  2025cites this paper
• Optimal sparsity in autoencoder memory models of the hippocampus

  2025cites this paper
• Concept Steerers: Leveraging K-Sparse Autoencoders for Test-Time Controllable Generations

  2025influential citation
• Discovering and Steering Interpretable Concepts in Large Generative Music Models

  2025cites this paper
• TIDE : Temporal-Aware Sparse Autoencoders for Interpretable Diffusion Transformers in Image Generation

  2025cites this paper
• Neural Network‐Based Study for Rice Leaf Disease Recognition and Classification: A Comparative Analysis Between Feature‐Based Model and Direct Imaging Model

  2025cites this paper
• Interpreting CFD Surrogates through Sparse Autoencoders

  2025cites this paper
• Infusing structural assumptions into dimensionality reduction for single-cell RNA sequencing data to identify small gene sets

  2025cites this paper
• SplInterp: Improving our Understanding and Training of Sparse Autoencoders

  2025cites this paper
• A Deep Learning-Based Method for Mechanical Equipment Unknown Fault Detection in the Industrial Internet of Things

  2025cites this paper
• Sparse Autoencoders for Hypothesis Generation

  2025cites this paper
• Diversity-driven Data Selection for Language Model Tuning through Sparse Autoencoder

  2025cites this paper
• Self-Regularization with Sparse Autoencoders for Controllable LLM-based Classification

  2025influential citation
• Interpretable and Testable Vision Features via Sparse Autoencoders

  2025cites this paper
• Multi-Faceted Multimodal Monosemanticity

  2025cites this paper
• Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representation

  2025cites this paper
• Interpreting and Steering LLMs with Mutual Information-based Explanations on Sparse Autoencoders

  2025cites this paper
• Steered Generation via Gradient Descent on Sparse Features

  2025cites this paper
• Mixture of Experts Made Intrinsically Interpretable

  2025cites this paper
• Privacy-Diffusion: Privacy-Preserving Stable Diffusion Without FHE and Differential Privacy

  2025cites this paper
• Route Sparse Autoencoder to Interpret Large Language Models

  2025cites this paper
• Nonlinear Multiple Response Regression and Learning of Latent Spaces

  2025cites this paper
• Task-Specific Data Selection for Instruction Tuning via Monosemantic Neuronal Activations

  2025cites this paper
• Model-Agnostic Gender Bias Control for Text-to-Image Generation via Sparse Autoencoder

  2025cites this paper
• PrivacyScalpel: Enhancing LLM Privacy via Interpretable Feature Intervention with Sparse Autoencoders

  2025influential citation
• Sparse Autoencoder as a Zero-Shot Classifier for Concept Erasing in Text-to-Image Diffusion Models

  2025cites this paper
• Sparse Autoencoders Learn Monosemantic Features in Vision-Language Models

  2025cites this paper
• Interpreting the Linear Structure of Vision-language Model Embedding Spaces

  2025cites this paper
• Evaluating and Designing Sparse Autoencoders by Approximating Quasi-Orthogonality

  2025cites this paper
• Compatible Unsupervised Anomaly Detection with Multi-Perspective Spatio-Temporal Learning

  2025cites this paper
• Train One Sparse Autoencoder Across Multiple Sparsity Budgets to Preserve Interpretability and Accuracy

  2025cites this paper
• K-sparse encoder for efficient information retrieval

  2025cites this paper
• End-to-End Neural Video Compression: A Review

  2025cites this paper
• A Real-time Anomaly Detection Method for Robots based on a Flexible and Sparse Latent Space

  2025cites this paper
• The Future is Sparse: Embedding Compression for Scalable Retrieval in Recommender Systems

  2025cites this paper
• Evaluating Sparse Autoencoders: From Shallow Design to Matching Pursuit

  2025cites this paper
• Explanations Go Linear: Post-Hoc Explainability for Tabular Data with Interpretable Meta-Encoding

  2025cites this paper
• Low-Rank Adapting Models for Sparse Autoencoders

  2025cites this paper
• Base Models Know How to Reason, Thinking Models Learn When

  2025cites this paper
• Concept-Guided Interpretability via Neural Chunking

  2025cites this paper