Continual Learning in Linear Classification on Separable Data

We analyze continual learning on a sequence of separable linear classification tasks with binary labels. We show theoretically that learning with weak regularization reduces to solving a sequential max-margin problem, corresponding to a special case of the Projection Onto Convex Sets (POCS) framework. We then develop upper bounds on the forgetting and other quantities of interest under various settings with recurring tasks, including cyclic and random orderings of tasks. We discuss several practical implications to popular training practices like regularization scheduling and weighting. We point out several theoretical differences between our continual classification setting and a recently studied continual regression setting.

• Publication year

  2023

• Venue

  International Conference on Machine Learning

• Publication date

  2023-06-06

• Fields of study

  Mathematics, Computer Science

• Identifiers
  DOI 10.48550/arXiv.2306.03534arXiv 2306.03534
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• The Ideal Continual Learner: An Agent That Never Forgets

  2023cited by this paper
• Is forgetting less a good inductive bias for forward transfer?

  2023influential reference
• Fixed Design Analysis of Regularization-Based Continual Learning

  2023cited by this paper
• A Closer Look at Rehearsal-Free Continual Learning *

  2022cited by this paper
• How catastrophic can catastrophic forgetting be in linear regression?

  2022influential reference
• Polynomial estimates for the method of cyclic projections in Hilbert spaces

  2022cited by this paper
• Scaling the Number of Tasks in Continual Learning

  2022cited by this paper
• Analysis of Catastrophic Forgetting for Random Orthogonal Transformation Tasks in the Overparameterized Regime

  2022cited by this paper
• Using Domain Adaptation for Incremental SVM Classification of Drift Data

  2022cited by this paper
• Continual Learning in the Teacher-Student Setup: Impact of Task Similarity

  2021cited by this paper
• Is Class-Incremental Enough for Continual Learning?

  2021cited by this paper
• Statistical Mechanical Analysis of Catastrophic Forgetting in Continual Learning with Teacher and Student Networks

  2021cited by this paper
• How do Quadratic Regularizers Prevent Catastrophic Forgetting: The Role of Interpolation

  2021cited by this paper
• Generalisation Guarantees for Continual Learning with Orthogonal Gradient Descent

  2020cited by this paper
• On the training dynamics of deep networks with L2 regularization

  2020cited by this paper
• A Theoretical Analysis of Catastrophic Forgetting through the NTK Overlap Matrix

  2020cited by this paper
• Understanding the Role of Training Regimes in Continual Learning

  2020cited by this paper
• Sampling Kaczmarz-Motzkin method for linear feasibility problems: generalization and acceleration

  2020influential reference
• Kernel and Rich Regimes in Overparametrized Models

  2019cited by this paper
• Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networks

  2019cited by this paper
• MS & E 213 / CS 269 O Chapter 3-Convexity ∗

  2019cited by this paper
• Incremental Object Learning From Contiguous Views

  2019cited by this paper
• Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernel

  2018influential reference
• Online Learning: A Comprehensive Survey

  2018cited by this paper
• A Continuous-Time View of Early Stopping for Least Squares Regression

  2018cited by this paper
• Gradient Descent Finds Global Minima of Deep Neural Networks

  2018cited by this paper
• Experience Replay for Continual Learning

  2018cited by this paper
• To understand deep learning we need to understand kernel learning

  2018cited by this paper
• Memory Aware Synapses: Learning what (not) to forget

  2017cited by this paper
• Lifelong Learning with Dynamically Expandable Networks

  2017cited by this paper
• The Implicit Bias of Gradient Descent on Separable Data

  2017influential reference
• Continual Learning Through Synaptic Intelligence

  2017influential reference
• Overcoming catastrophic forgetting in neural networks

  2016influential reference
• Curriculum learning of multiple tasks

  2014cited by this paper
• Analysis of the Convergence Rate for the Cyclic Projection Algorithm Applied to Basic Semialgebraic Convex Sets

  2013cited by this paper
• An Empirical Investigation of Catastrophic Forgeting in Gradient-Based Neural Networks

  2013influential reference
• Paved with Good Intentions: Analysis of a Randomized Block Kaczmarz Method

  2012cited by this paper
• Online Learning and Online Convex Optimization

  2012cited by this paper
• Stochastic Gradient Descent for Non-smooth Optimization: Convergence Results and Optimal Averaging Schemes

  2012cited by this paper
• Convex Analysis and Monotone Operator Theory in Hilbert Spaces

  2011influential reference
• Convex optimization

  2010influential reference
• Random projection algorithms for convex set intersection problems

  2010influential reference
• The rate of convergence for the cyclic projections algorithm III: Regularity of convex sets

  2008influential reference
• Regularized adaptation: theory, algorithms and applications

  2007influential reference
• Nanyang Technological University School of Electrical and Electronic Engineering Republic of Singapore

  2007influential reference
• The rate of convergence for the cyclic projections algorithm II: Norms of nonlinear operators

  2006influential reference
• The rate of convergence for the cyclic projections algorithm I: Angles between convex sets

  2006influential reference
• Margin Maximizing Loss Functions

  2003influential reference
• Online Passive-Aggressive Algorithms

  2003cited by this paper
• Projection Algorithms: Results and Open Problems

  2001influential reference
• Catastrophic Forgetting, Rehearsal and Pseudorehearsal

  1995cited by this paper
• On the convergence of von Neumann's alternating projection algorithm for two sets

  1993influential reference
• Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem

  1989cited by this paper
• The method of projections for finding the common point of convex sets

  1967influential reference
• ON CONVERGENCE PROOFS FOR PERCEPTRONS

  1963cited by this paper
• The Relaxation Method for Linear Inequalities

  1954cited by this paper
• On Rings of Operators. Reduction Theory

  1949cited by this paper

• Putting a Face to Forgetting: Continual Learning meets Mechanistic Interpretability

  2026cites this paper
• Understanding the Role of Rehearsal Scale in Continual Learning under Varying Model Capacities

  2026cites this paper
• Global Convergence of Continual Learning on Non-IID Data

  2025cites this paper
• From Continual Learning to SGD and Back: Better Rates for Continual Linear Models

  2025influential citation
• Unlocking the Power of Rehearsal in Continual Learning: A Theoretical Perspective

  2025cites this paper
• Replay Can Provably Increase Forgetting

  2025cites this paper
• Optimal Rates in Continual Linear Regression via Increasing Regularization

  2025influential citation
• Fast Last-Iterate Convergence of SGD in the Smooth Interpolation Regime

  2025cites this paper
• A statistical physics framework for optimal learning

  2025cites this paper
• On Understanding of the Dynamics of Model Capacity in Continual Learning

  2025influential citation
• Intelligent Fault Diagnosis for Rotating Machinery via Transfer Learning and Attention Mechanisms: A Lightweight and Adaptive Approach

  2025cites this paper
• On the Theory of Continual Learning with Gradient Descent for Neural Networks

  2025cites this paper
• The Unreasonable Effectiveness of Randomized Representations in Online Continual Graph Learning

  2025cites this paper
• Data Heterogeneity and Forgotten Labels in Split Federated Learning

  2025cites this paper
• Optimal Task Order for Continual Learning of Multiple Tasks

  2025cites this paper
• Analysis of Overparameterization in Continual Learning under a Linear Model

  2025cites this paper
• N3C: Towards Replay-based Novelty Continual Clustering with Class-Overlapping

  2025cites this paper
• Memory-Statistics Tradeoff in Continual Learning with Structural Regularization

  2025cites this paper
• A Statistical Theory of Regularization-Based Continual Learning

  2024cites this paper
• Adaptive Margin Global Classifier for Exemplar-Free Class-Incremental Learning

  2024cites this paper
• Disentangling and Mitigating the Impact of Task Similarity for Continual Learning

  2024cites this paper
• Last Iterate Convergence of Incremental Methods and Applications in Continual Learning

  2024influential citation
• Similarity-Based Adaptation for Task-Aware and Task-Free Continual Learning

  2024cites this paper
• On the Disconnect Between Theory and Practice of Overparametrized Neural Networks

  2023cites this paper
• Are deep networks broken, and should we fix them?

  2023cites this paper
• On the Disconnect Between Theory and Practice of Neural Networks: Limits of the NTK Perspective

  2023cites this paper
• Inductive biases of multi-task learning and finetuning: multiple regimes of feature reuse

  2023cites this paper
• The Ideal Continual Learner: An Agent That Never Forgets

  2023cites this paper
• A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learning

  2023cites this paper