Asymptotic Distribution of Stochastic Mirror Descent Iterates in Average Ensemble Models

The stochastic mirror descent (SMD) algorithm is a general class of training algorithms that utilizes a mirror potential to influence the implicit bias of the training algorithm and includes stochastic gradient descent (SGD) as a special case. In this paper, we explore the performance of the SMD on mean-field ensemble models and generalize earlier results obtained for SGD. The evolution of the distribution of parameters is mapped to a continuous time process in the space of probability distributions. Our main result gives a nonlinear partial differential equation (PDE) to which the continuous time process converges in the asymptotic of large networks. The impact of the mirror potential appears through a multiplicative term that is equal to the inverse of its Hessian and defines a gradient flow over an appropriate Riemannian manifold. We provide numerical simulations which allow us to study and characterize the effect of the mirror potential on the performance of networks trained with SMD for some binary classification problems.

• Publication year

  2023

• Venue

  IEEE International Conference on Acoustics, Speech, and Signal Processing

• Publication date

  2023-06-04

• Fields of study

  Mathematics, Computer Science

• Identifiers
  DOI 10.1109/ICASSP49357.2023.10096701
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Stochastic Mirror Descent in Average Ensemble Models

  2022influential reference
• Mirrorless Mirror Descent: A More Natural Discretization of Riemannian Gradient Flow

  2020cited by this paper
• On stochastic mirror descent with interacting particles: Convergence properties and variance reduction

  2020cited by this paper
• Stochastic Mirror Descent on Overparameterized Nonlinear Models

  2019cited by this paper
• A Stochastic Interpretation of Stochastic Mirror Descent: Risk-Sensitive Optimality

  2019cited by this paper
• Mean Field Analysis of Neural Networks: A Law of Large Numbers

  2018cited by this paper
• Characterizing Implicit Bias in Terms of Optimization Geometry

  2018cited by this paper
• A mean field view of the landscape of two-layer neural networks

  2018cited by this paper
• On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport

  2018cited by this paper
• Scaling Limit: Exact and Tractable Analysis of Online Learning Algorithms with Applications to Regularized Regression and PCA

  2017cited by this paper
• Deep Learning

  2015cited by this paper
• Convex Optimization: Algorithms and Complexity

  2014cited by this paper
• The Information Geometry of Mirror Descent

  2013cited by this paper
• ImageNet classification with deep convolutional neural networks

  2012cited by this paper
• Continuous-time stochastic Mirror Descent on a network: Variance reduction, consensus, convergence

  2012cited by this paper
• Optimal Transport: Old and New

  2008cited by this paper
• Problem Complexity and Method Efficiency in Optimization

  1983cited by this paper
• Convergence of probability measures

  year unknowncited by this paper

• No citing papers are available for this paper.