Learning Step Size Controllers for Robust Neural Network Training

This paper investigates algorithms to automatically adapt the learning rate of neural networks (NNs). Starting with stochastic gradient descent, a large variety of learning methods has been proposed for the NN setting. However, these methods are usually sensitive to the initial learning rate which has to be chosen by the experimenter. We investigate several features and show how an adaptive controller can adjust the learning rate without prior knowledge of the learning problem at hand.

• Publication year

  2016

• Venue

  AAAI Conference on Artificial Intelligence

• Publication date

  2016-02-12

• Fields of study

  Computer Science

• Identifiers
  DOI 10.1609/aaai.v30i1.10187
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• RMSProp and equilibrated adaptive learning rates for non-convex optimization

  2015cited by this paper
• MatConvNet: Convolutional Neural Networks for MATLAB

  2014cited by this paper
• Adam: A Method for Stochastic Optimization

  2014cited by this paper
• Learning sequential motor tasks

  2013cited by this paper
• An empirical study of learning rates in deep neural networks for speech recognition

  2013cited by this paper
• Practical Bayesian Optimization of Machine Learning Algorithms

  2012cited by this paper
• No more pesky learning rates

  2012cited by this paper
• Comunication-Efficient Algorithms for Statistical Optimization

  2012cited by this paper
• ADADELTA: An Adaptive Learning Rate Method

  2012cited by this paper
• Communication-efficient algorithms for statistical optimization

  2012cited by this paper
• Adaptive Subgradient Methods for Online Learning and Stochastic Optimization

  2011cited by this paper
• Learning Dynamic Tactile Sensing With Robust Vision-Based Training

  2011cited by this paper
• A Simplified Natural Gradient Learning Algorithm

  2011cited by this paper
• Improving First and Second-Order Methods by Modeling Uncertainty

  2010cited by this paper
• Relative Entropy Policy Search

  2010influential reference
• Building Controllers for Tetris

  2009cited by this paper
• Advances in Neural Information Processing Systems (NIPS)

  2007cited by this paper
• Reinforcement Learning: An Introduction

  1998cited by this paper
• Introduction to Reinforcement Learning

  1998cited by this paper
• Natural Gradient Works Efficiently in Learning

  1998cited by this paper
• A direct adaptive method for faster backpropagation learning: the RPROP algorithm

  1993cited by this paper
• Adapting Bias by Gradient Descent: An Incremental Version of Delta-Bar-Delta

  1992cited by this paper
• Fast adaptive k-means clustering: some empirical results

  1990cited by this paper
• Increased rates of convergence through learning rate adaptation

  1987cited by this paper

• A general learning rate improvement strategy for deep neural networks training

  2026cites this paper
• Hybrid Reinforcement Learning to Optimize for Physics-Constrained Spectral Analysis

  2025cites this paper
• Let the Optimizers Optimize Themselves

  2025cites this paper
• Sequential Multi-Agent Dynamic Algorithm Configuration

  2025cites this paper
• RLGS: Reinforcement Learning-Based Adaptive Hyperparameter Tuning for Gaussian Splatting

  2025cites this paper
• Revisiting Learning Rate Control

  2025cites this paper
• Sequential Policy Gradient for Adaptive Hyperparameter Optimization

  2025cites this paper
• Dynamic Algorithmic Configuration for Enhanced Malware Detection

  2025cites this paper
• Meta-Black-Box optimization for evolutionary algorithms: Review and perspective

  2025cites this paper
• Communication-Efficient Network Topology in Decentralized Learning: A Joint Design of Consensus Matrix and Resource Allocation

  2025cites this paper
• Artificial neural network‐based adaptive control for nonlinear dynamical systems

  2024cites this paper
• A Novel Paradigm for Neural Computation: X-Net with Learnable Neurons and Adaptable Structure

  2024cites this paper
• Designing L-BFGS inspired automatic optimizer network

  2024cites this paper
• Narrowing the Focus: Learned Optimizers for Pretrained Models

  2024cites this paper
• Accelerate Evolution Strategy by Proximal Policy Optimization

  2024cites this paper
• Generative Models of Images and Neural Networks

  2023cites this paper
• Learning to Learn Task-Adaptive Hyperparameters for Few-Shot Learning

  2023cites this paper
• Performance Optimization for Noise Interference Privacy Protection in Federated Learning

  2023cites this paper
• Intelligent gradient amplification for deep neural networks

  2023cites this paper
• Theory-inspired parameter control benchmarks for dynamic algorithm configuration

  2022cites this paper
• Learning an Interpretable Learning Rate Schedule via the Option Framework

  2022cites this paper
• AIADA: Accuracy Impact Assessment of Deprecated Python API Usages on Deep Learning Models

  2022cites this paper
• Implicit Training of Energy Model for Structure Prediction

  2022cites this paper
• VeLO: Training Versatile Learned Optimizers by Scaling Up

  2022cites this paper
• Multi-agent Dynamic Algorithm Configuration

  2022cites this paper
• Balancing the Stability-Plasticity Dilemma with Online Stability Tuning for Continual Learning

  2022cites this paper
• Learning to Learn with Generative Models of Neural Network Checkpoints

  2022cites this paper
• A Closer Look at Learned Optimization: Stability, Robustness, and Inductive Biases

  2022cites this paper
• Learning to Schedule Learning rate with Graph Neural Networks

  2022cites this paper
• A sharp uniform-in-time error estimate for Stochastic Gradient Langevin Dynamics

  2022cites this paper
• On uniform-in-time diffusion approximation for stochastic gradient descent

  2022cites this paper
• Automated Dynamic Algorithm Configuration

  2022influential citation
• Chaining of Numerical Black-box Algorithms: Warm-Starting and Switching Points

  2022cites this paper
• Practical tradeoffs between memory, compute, and performance in learned optimizers

  2022influential citation
• Automated Reinforcement Learning: An Overview

  2022cites this paper
• Efficient Meta Subspace Optimization

  2021cites this paper
• Convolutional Neural Network With Automatic Learning Rate Scheduler for Fault Classification

  2021influential citation
• DACBench: A Benchmark Library for Dynamic Algorithm Configuration

  2021influential citation
• Awesome back-propagation machine learning paradigm

  2021cites this paper
• A Generalizable Approach to Learning Optimizers

  2021cites this paper
• To raise or not to raise: the autonomous learning rate question

  2021influential citation
• Deep Learning for Visual Analytics of the Spread of COVID-19 Infection in Crowded Urban Environments

  2021cites this paper
• Meta-Optimization of Deep CNN for Image Denoising Using LSTM

  2021cites this paper
• Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges

  2021cites this paper
• The Effect of Adaptive Learning Rate on the Accuracy of Neural Networks

  2021cites this paper
• Meta Subspace Optimization

  2021cites this paper
• Efficient Learning to Learn a Robust CTR Model for Web-scale Online Sponsored Search Advertising

  2021cites this paper
• Learning Step-Size Adaptation in CMA-ES

  2020cites this paper
• O VERCOMING BARRIERS TO THE TRAINING OF EFFEC - TIVE LEARNED OPTIMIZERS

  2020cites this paper
• Advances in Deep Learning through Gradient Amplification and Applications

  2020cites this paper
• Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves

  2020influential citation
• On the step size selection in variance-reduced algorithm for nonconvex optimization

  2020cites this paper
• Learning the Step-size Policy for the Limited-Memory Broyden-Fletcher-Goldfarb-Shanno Algorithm

  2020cites this paper
• Dynamic Algorithm Configuration: Foundation of a New Meta-Algorithmic Framework

  2020cites this paper
• Training more effective learned optimizers

  2020cites this paper
• Learning an Adaptive Learning Rate Schedule

  2019cites this paper
• Combining Model-Based with Learning-Based Approaches for Autonomous Manipulation

  2019influential citation
• Meta-Learner with Sparsified Backpropagation

  2019influential citation
• Towards White-box Benchmarks for Algorithm Control

  2019cites this paper
• Convolutional neural network architecture study for aerial visual localization

  2019cites this paper
• Meta-Learning Acquisition Functions for Bayesian Optimization

  2019cites this paper
• Mini-batch algorithms with online step size

  2019cites this paper
• Advances in Machine Learning: Nearest Neighbour Search, Learning to Optimize and Generative Modelling

  2019cites this paper
• Robust Federated Learning Through Representation Matching and Adaptive Hyper-parameters

  2019cites this paper
• Fixing Implicit Derivatives: Trust-Region Based Learning of Continuous Energy Functions

  2019cites this paper
• On ADMM in Deep Learning: Convergence and Saturation-Avoidance

  2019cites this paper
• Meta-Learning Acquisition Functions for Transfer Learning in Bayesian Optimization

  2019cites this paper
• HyperAdam: A Learnable Task-Adaptive Adam for Network Training

  2018cites this paper
• MQGrad: Reinforcement Learning of Gradient Quantization in Parameter Server

  2018cites this paper
• Effective neural network training with adaptive learning rate based on training loss

  2018cites this paper
• Rover Descent: Learning to optimize by learning to navigate on prototypical loss surfaces

  2018cites this paper
• Gradient Descent is a Technique for Learning to Learn

  2018cites this paper
• Training Neural Networks Using Predictor-Corrector Gradient Descent

  2018cites this paper
• Worker 1 gradient modelserver Worker P gradient model 1 2 2

  2018cites this paper
• Distributed algorithms for optimized resource management of LTE in unlicensed spectrum and UAV-enabled wireless networks

  2018cites this paper
• Meta-Learning: A Survey

  2018cites this paper
• Decentralized Differentially Private Without-Replacement Stochastic Gradient Descent

  2018cites this paper
• Exploratory model analysis for machine learning

  2018cites this paper
• Step Size Matters in Deep Learning

  2018cites this paper
• An Actor-critic Algorithm for Learning Rate Learning

  2017cites this paper
• Learning to Optimize Neural Nets

  2017cites this paper
• Variants of RMSProp and Adagrad with Logarithmic Regret Bounds

  2017cites this paper
• Online Learning of a Memory for Learning Rates

  2017cites this paper
• Leukocytes Classification and Segmentation in Microscopic Blood Smear: A Resource-Aware Healthcare Service in Smart Cities

  2017cites this paper
• Learning Gradient Descent: Better Generalization and Longer Horizons

  2017cites this paper
• Interactive Machine Learning via a GPU-accelerated Toolkit

  2017cites this paper
• Proactive Resource Management for LTE in Unlicensed Spectrum: A Deep Learning Perspective

  2017cites this paper
• Reinforcement Learning for Learning Rate Control

  2017cites this paper
• Proactive Resource Management in LTE-U Systems: A Deep Learning Perspective

  2017cites this paper
• ING OF DEEP NEURAL NETWORKS

  2016influential citation
• Learning to learn by gradient descent by reinforcement learning

  2016cites this paper
• Deep Reinforcement Learning for Accelerating the Convergence Rate

  2016influential citation
• Learning to learn by gradient descent by gradient descent

  2016cites this paper
• Learning to Draw Samples: With Application to Amortized MLE for Generative Adversarial Learning

  2016cites this paper
• Deep Q-Networks for Accelerating the Training of Deep Neural Networks

  2016influential citation
• Communication-Efficient Network Topology in Decentralized Learning: A Joint Design of Consensus Matrix and Resource Allocation

  year unknowncites this paper
• Command the Flow: Analysis and Prediction of Momentum in Tennis Matches

  year unknowncites this paper