Online estimation of the asymptotic variance for averaged stochastic gradient algorithms

Stochastic gradient algorithms are more and more studied since they can deal efficiently and online with large samples in high dimensional spaces. In this paper, we first establish a Central Limit Theorem for these estimates as well as for their averaged version in general Hilbert spaces. Moreover, since having the asymptotic normality of estimates is often unusable without an estimation of the asymptotic variance, we introduce a new recursive algorithm for estimating this last one, and we establish its almost sure rate of convergence as well as its rate of convergence in quadratic mean. Finally, two examples consisting in estimating the parameters of the logistic regression and estimating geometric quantiles are given.

• Publication year

  2017

• Venue

  Journal of Statistical Planning and Inference

• Publication date

  2017-02-03

• Fields of study

  Mathematics, Computer Science

• Identifiers
  DOI 10.1016/J.JSPI.2019.01.001arXiv 1702.00931
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Robust Statistics

  2018cited by this paper
• An averaged projected Robbins-Monro algorithm for estimating the parameters of a truncated spherical distribution

  2016cited by this paper
• Estimating the geometric median in Hilbert spaces with stochastic gradient algorithms: Lp and almost sure rates of convergence

  2016influential reference
• $L^{p}$ and almost sure rates of convergence of averaged stochastic gradient algorithms with applications to online robust estimation

  2016influential reference
• Fast estimation of the median covariation matrix with application to online robust principal components analysis

  2015cited by this paper
• Online estimation of the geometric median in Hilbert spaces : non asymptotic confidence balls

  2015influential reference
• The spatial distribution in infinite dimensional spaces and related quantiles and depths

  2014cited by this paper
• Deterministic and Stochastic Primal-Dual Subgradient Algorithms for Uniformly Convex Minimization

  2014cited by this paper
• Scalable and Robust Bayesian Inference via the Median Posterior

  2014cited by this paper
• Adaptivity of averaged stochastic gradient descent to local strong convexity for logistic regression

  2013influential reference
• Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)

  2013cited by this paper
• Stochastic Approximation approach to Stochastic Programming

  2013cited by this paper
• Dispersion operators and resistant second-order functional data analysis

  2012cited by this paper
• Efficient and fast estimation of the geometric median in Hilbert spaces with an averaged stochastic gradient algorithm

  2011cited by this paper
• Convex optimization

  2010cited by this paper
• Robust functional estimation using the median and spherical principal components

  2008influential reference
• SEMIPARAMETRICALLY EFFICIENT RANK-BASED INFERENCE FOR SHAPE I. OPTIMAL RANK-BASED TESTS FOR SPHERICITY

  2006cited by this paper
• SEMIPARAMETRICALLY EFFICIENT RANK-BASED INFERENCE FOR SHAPE II. OPTIMAL R-ESTIMATION OF SHAPE

  2006cited by this paper
• The Averaged Robbins – Monro Method for Linear Problems in a Banach Space

  2006influential reference
• Stochastic Approximation and Recursive Algorithms and Applications

  2003cited by this paper
• The multivariate L1-median and associated data depth.

  2000cited by this paper
• On the estimation of the asymptotic covariance matrix for the averaged Robbins–Monro algorithm

  2000cited by this paper
• Asymptotic Almost Sure Efficiency of Averaged Stochastic Algorithms

  2000influential reference
• On the almost sure asymptotic behaviour of stochastic algorithms

  1998cited by this paper
• Weighted Means in Stochastic Approximation of Minima

  1997cited by this paper
• On a geometric notion of quantiles for multivariate data

  1996influential reference
• On a stochastic approximation procedure based on averaging

  1996cited by this paper
• Accelerated Stochastic Approximation

  1993cited by this paper
• Foundations of stochastic approximation

  1992cited by this paper
• Acceleration of stochastic approximation by averaging

  1992cited by this paper
• Stochastic optimization with averaging of trajectories

  1992cited by this paper
• Stochastic approximation and optimization of random systems

  1992cited by this paper
• Efficient Estimations from a Slowly Convergent Robbins-Monro Process

  1988cited by this paper
• TIGHTNESS CRITERIA FOR RANDOM MEASURES WITH APPLICATION TO THE PRINCIPLE OF CONDITIONING IN I-ULBEWT SPACES

  1988influential reference
• Asymptotic Properties of the Generalized Median in the Case of Multivariate Normality

  1985cited by this paper
• Matrix analysis

  1985influential reference
• On Asymptotic Normality in Stochastic Approximation

  1968cited by this paper
• A Stochastic Approximation Method

  1951cited by this paper
• Note on the median of a multivariate distribution

  1948cited by this paper
• DIMACS Series in Discrete Mathematics and Theoretical Computer Science Depth Functions in Nonparametric Multivariate Inference Robert Serfling

  year unknowncited by this paper

• Complexity reduction in online stochastic Newton methods with potential O(N d) total cost

  2026cites this paper
• Statistical Inference for Online Algorithms

  2025cites this paper
• Online estimation of the inverse of the Hessian for stochastic optimization with application to universal stochastic Newton algorithms

  2024cites this paper
• Online bootstrap inference for the geometric median

  2024cites this paper
• Online robust estimation and bootstrap inference for function-on-scalar regression

  2024cites this paper
• Online and offline robust multivariate linear regression

  2024influential citation
• An Online Bootstrap for Time Series

  2023cites this paper
• Online stochastic Newton methods for estimating the geometric median and applications

  2023cites this paper
• On Adaptive Stochastic Optimization for Streaming Data: A Newton's Method with O(dN) Operations

  2023cites this paper
• Statistical Estimation and Online Inference via Local SGD

  2022cites this paper
• RETRACTED ARTICLE: Convergence in quadratic mean of averaged stochastic gradient algorithms without strong convexity nor bounded gradient

  2021influential citation
• On the asymptotic rate of convergence of Stochastic Newton algorithms and their Weighted Averaged versions

  2020cites this paper
• Stochastic approximation algorithms for superquantiles estimation

  2020cites this paper
• An efficient averaged stochastic Gauss-Newton algorithm for estimating parameters of nonlinear regressions models

  2020cites this paper
• An Efficient Stochastic Newton Algorithm for Parameter Estimation in Logistic Regressions

  2019cites this paper
• On the rates of convergence of parallelized averaged stochastic gradient algorithms

  2017cites this paper