Effective neural network training with adaptive learning rate based on training loss

A method that uses an adaptive learning rate is presented for training neural networks. Unlike most conventional updating methods in which the learning rate gradually decreases during training, the proposed method increases or decreases the learning rate adaptively so that the training loss (the sum of cross-entropy losses for all training samples) decreases as much as possible. It thus provides a wider search range for solutions and thus a lower test error rate. The experiments with some well-known datasets to train a multilayer perceptron show that the proposed method is effective for obtaining a better test accuracy under certain conditions.

• Publication year

  2018

• Venue

  Neural Networks

• Publication date

  2018-05-01

• Fields of study

  Medicine, Computer Science

• Identifiers
  DOI 10.1016/j.neunet.2018.01.016PMID 29494873
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar, PubMed

• No claims are published for this paper.

• No concepts are published for this paper.

• Evaluating deep learning architectures for Speech Emotion Recognition

  2017cited by this paper
• Rank-based pooling for deep convolutional neural networks

  2016cited by this paper
• Optimizing neural-network learning rate by using a genetic algorithm with per-epoch mutations

  2016cited by this paper
• A Variational Analysis of Stochastic Gradient Algorithms

  2016cited by this paper
• Extreme Learning Machine for Multilayer Perceptron

  2016cited by this paper
• Approximate Bayesian MLP regularization for regression in the presence of noise

  2016cited by this paper
• Learning Step Size Controllers for Robust Neural Network Training

  2016cited by this paper
• Train faster, generalize better: Stability of stochastic gradient descent

  2015cited by this paper
• On the Convergence of SGD Training of Neural Networks

  2015cited by this paper
• Path-SGD: Path-Normalized Optimization in Deep Neural Networks

  2015cited by this paper
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

  2015cited by this paper
• Training Deep Neural Networks via Direct Loss Minimization

  2015cited by this paper
• Towards dropout training for convolutional neural networks

  2015cited by this paper
• Deep Convolutional Neural Networks for Large-scale Speech Tasks

  2015cited by this paper
• Dropout: a simple way to prevent neural networks from overfitting

  2014cited by this paper
• Adam: A Method for Stochastic Optimization

  2014cited by this paper
• Accelerating recurrent neural network training via two stage classes and parallelization

  2013cited by this paper
• Efficient BackProp

  2012cited by this paper
• ADADELTA: An Adaptive Learning Rate Method

  2012influential reference
• ImageNet classification with deep convolutional neural networks

  2012cited by this paper
• Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups

  2012cited by this paper
• Deep Sparse Rectifier Neural Networks

  2011cited by this paper
• Adaptive Subgradient Methods for Online Learning and Stochastic Optimization

  2011cited by this paper
• Understanding the difficulty of training deep feedforward neural networks

  2010cited by this paper
• The mnist database of handwritten digits

  2005cited by this paper
• Rank, Trace-Norm and Max-Norm

  2005cited by this paper
• Reinforcement Learning: An Introduction

  1998cited by this paper
• Assessing the importance of features for multi-layer perceptrons

  1998cited by this paper
• Gradient-based learning applied to document recognition

  1998cited by this paper

• Design of trailing edge flap based on MFC and prediction of deformation and aerodynamic performance using BP neural network

  2025cites this paper
• Parametric Optimization of Artificial Neural Networks and Machine Learning Techniques Applied to Small Welding Datasets

  2025cites this paper
• Recent Progress in Memristor Array‐Based Neuromorphic Computing for on‐Chip Vector‐Matrix Multiplication

  2025cites this paper
• Comparative analysis of deep learning architectures for multi-class mineral classification: a study using EfficientNet and ResNet models

  2025cites this paper
• Prediction of Cerebrospinal Fluid (CSF) Pressure with Generative Adversarial Network Synthetic Plasma-CSF Biomarker Pairing

  2025cites this paper
• In vitro temperature-dependent degradation of polylactic acid: Experimental characterization and neural network-based prediction

  2025cites this paper
• Implementation and Evaluation of Artificial Neural Networks for Product Sales Prediction at Basmalah Stores

  2025cites this paper
• Robust S3Former deep learning model for the direct diagnosis and prediction of natural organic matter (NOM) from three-dimensional excitation-emission-matrix (3D-EEM) data.

  2025cites this paper
• Predicting and Analyzing the Immobilization of MgAl and CaAl Layered Double Hydroxides for Cr(VI) in Polluted Soil Using Machine Learning

  2025cites this paper
• Artificial Neural Networks for Estimating Historical Daily Missing Evaporation to Support Sustainable Development in Saudi Arabia

  2025cites this paper
• Professor X: Diagnosis and Treatment of Dermatological Diseases by Integration of Visual Diagnosis and Retrieval-Augmented Generation (RAG) Technologies

  2025cites this paper
• Karcı kesir dereceli yapay sinir ağı (KarcıFANN): öğrenme oranı, aşırı uyum ve yetersiz uyum sorunlarını çözme

  2025cites this paper
• Prediction of Mechanical Properties of Weathered Injection-Molded Polycarbonate/Acrylonitrile–Butadiene–Styrene Blends using Fourier-Transform Infrared Spectroscopy–based 1D Convolutional Neural Network

  2025cites this paper
• Development of a Deep Neural Network Controller with an Adaptive Learning Factor: A Case Study on Position Control of a Stepper Motor

  2025cites this paper
• GhostPointNet: A deep learning-based method for ghost point noise detection in four-dimensional (4D) millimeter-wave radar point clouds of underground mine

  2025cites this paper
• Hyperparameter optimization of apple leaf dataset for the disease recognition based on the YOLOv8

  2025cites this paper
• Audio-visual self-supervised representation learning: A survey

  2025cites this paper
• Hyperparameter Optimization for Tomato Leaf Disease Recognition Based on YOLOv11m

  2025cites this paper
• Artificial neural network controlled DSTATCOM for mitigating power quality concerns in solar PV and wind system

  2025cites this paper
• Artificial Neural Network-Driven Techno-Economic Predictions for Micro Gas Turbines (MGT) Based Energy Applications

  2025cites this paper
• Joint Transmission Mode Selection and Scheduling for AoI Minimization in NOMA-Capable WP-IoT Networks: A Deep Transfer Learning Solution

  2025cites this paper
• MLAPW: A framework to assess the impact of feature selection and sampling techniques on anti-pattern prediction using WSDL metrics

  2025cites this paper
• Deep neural network and transfer learning for annual wetland vegetation mapping using sentinel-2 time-series data in the heterogeneous lake floodplain environment

  2025cites this paper
• Automatic Detection of Banana Maturity—Application of Image Recognition in Agricultural Production

  2024cites this paper
• Efficient adaptive learning rate for convolutional neural network based on quadratic interpolation egret swarm optimization algorithm

  2024influential citation
• Performance of Deep Feed-Forward Neural Network Algorithm Based on Content-Based Filtering Approach

  2024cites this paper
• Modelling child comprehension: A case of suffixal passive construction in Korean

  2024cites this paper
• Normalization Layer Enhancement in Convolutional Neural Network for Parking Space Classification

  2024cites this paper
• Predicting prolonged hospital stay following hip dysplasia surgery in the pediatric population

  2024cites this paper
• Smart systems for real-time bearing faults diagnosis by using vibro-acoustics sensor fusion with Bayesian optimised 1-D CNNs

  2024cites this paper
• Engineering of TiN/ZnO/SnO2/ZnO/Pt multilayer memristor with advanced electronic synapses and analog switching for neuromorphic computing

  2024cites this paper
• Artificial neural network‐based adaptive control for nonlinear dynamical systems

  2024cites this paper
• Analysis of Machine Learning Algorithms for Prediction of Short-Term Rainfall Amounts Using Uganda’s Lake Victoria Basin Weather Dataset

  2024cites this paper
• An analytical approach for unsupervised learning rate estimation using rectified linear units

  2024cites this paper
• Fully automated whole brain segmentation from rat MRI scans with a convolutional neural network.

  2024cites this paper
• Inference of Network Connectivity from Temporally Binned Spike Trains.

  2024cites this paper
• Harnessing attention mechanisms in a comprehensive deep learning approach for induction motor fault diagnosis using raw electrical signals

  2024cites this paper
• Keep It Simple: Fault Tolerance Evaluation of Federated Learning with Unreliable Clients

  2023cites this paper
• Deep Multilayer Neural Network with Weights Optimization-Based Genetic Algorithm for Predicting Hypothyroid Disease

  2023cites this paper
• Adaptive orthogonal gradient descent algorithm for fully complex-valued neural networks

  2023cites this paper
• Sample-based Dynamic Hierarchical Transformer with Layer and Head Flexibility via Contextual Bandit

  2023cites this paper
• Streamline video-based automatic fabric pattern recognition using Bayesian-optimized convolutional neural network

  2023cites this paper
• Pengembangan Sistem Deteksi Otomatis FOD dengan YOLOv5 di Lingkungan Landasan Bandara

  2023cites this paper
• Enhanced Neural Network for Rapid Identification of Crop Water and Nitrogen Content Using Multispectral Imaging

  2023cites this paper
• Nonstationary significant wave height forecasting with a hybrid VMD-CNN model

  2023cites this paper
• Machine Learning-Based Automatic Litter Detection and Classification Using Neural Networks in Smart Cities

  2023cites this paper
• Dictionary-enabled efficient training of ConvNets for image classification

  2023cites this paper
• From micro- to nano- and time-resolved x-ray computed tomography: Bio-based applications, synchrotron capabilities, and data-driven processing

  2023cites this paper
• AdaSAM: Boosting Sharpness-Aware Minimization with Adaptive Learning Rate and Momentum for Training Deep Neural Networks

  2023cites this paper
• A Prediction Model for Spot LNG Prices Based on Machine Learning Algorithms to Reduce Fluctuation Risks in Purchasing Prices

  2023cites this paper
• DA-FER: Domain Adaptive Facial Expression Recognition

  2023cites this paper
• Machine learning with word embedding for detecting web-services anti-patterns

  2023cites this paper
• Penalized Least Squares Classifier: Classification by Regression Via Iterative Cost-Sensitive Learning

  2023cites this paper
• Classification of seed corn ears based on custom lightweight convolutional neural network and improved training strategies

  2023cites this paper
• ESOA Algorithm Based on learning rate optimization in Convolutional neural networks

  2023cites this paper
• Biomedical Image Classification via Dynamically Early Stopped Artificial Neural Network

  2022cites this paper
• Momentum-driven adaptive synchronization model for distributed DNN training on HPC clusters

  2022cites this paper
• The Era of Advanced Machine Learning and Deep Learning Algorithms for Malware Detection

  2022cites this paper
• A Review of Deep Transfer Learning and Recent Advancements

  2022cites this paper
• Construction of Feedforward Multilayer Perceptron Model for Diagnosing Leishmaniasis Using Transcriptome Datasets and Cognitive Computing

  2022cites this paper
• Second-order hyperparameter tuning of model-based and adaptive observers for time-varying and unknown chaotic systems

  2022cites this paper
• Development of an evolutionary artificial neural network-based tool for selecting suitable enhanced oil recovery methods

  2022cites this paper
• Off-State Performance Characterization of an AlGaN/GaN Device via Artificial Neural Networks

  2022cites this paper
• A Learning‐Rate Modulable and Reliable TiO x Memristor Array for Robust, Fast, and Accurate Neuromorphic Computing

  2022cites this paper
• Artificial Neural Network Analysis for Cost Estimation of Building Projects in India

  2022cites this paper
• Complex-Valued Neural Networks: A Comprehensive Survey

  2022cites this paper
• Artificial neural network for predicting annual output energy of building-integrated photovoltaics based on the 2-terminal perovskite/silicon tandem cells under realistic conditions

  2022cites this paper
• Inference of Presynaptic Connectivity from Temporally Blurry Spike Trains by Supervised Learning

  2022cites this paper
• FPGA implementation of a Convolutional Neural Network for image classification

  2022cites this paper
• Optimizing NARX-RNN Performance to Predict Precious Metal Futures market

  2022cites this paper
• EXPANSE: A Continual and Progressive Learning System for Deep Transfer Learning

  2022cites this paper
• Neural network prediction of 30-day mortality following primary total hip arthroplasty.

  2021cites this paper
• A Zeroth-Order Adaptive Learning Rate Method to Reduce Cost of Hyperparameter Tuning for Deep Learning

  2021influential citation
• Autism Spectrum Diagnosis using Adaptive Learning Algorithm for Multiple MLP Classifier

  2021cites this paper
• Compensation Network for Temperature Dependency and Bandwidth Expanding of Magnetoelectric Velocity Sensor Based on Phase Prediction Compensator

  2021cites this paper
• An improved convolutional neural network with an adaptable learning rate towards multi-signal fault diagnosis of hydraulic piston pump

  2021cites this paper
• Weighted neural tangent kernel: a generalized and improved network-induced kernel

  2021cites this paper
• Pure electric vehicle nonstationary interior sound quality prediction based on deep CNNs with an adaptable learning rate tree

  2021cites this paper
• EXPANSE, A Continual Deep Learning System; Research Proposal

  2021cites this paper
• Research on Precipitation Forecast Based on LSTM–CP Combined Model

  2021influential citation
• Beyond site suitability: Investigating temporal variability for utility-scale solar-PV using soft computing techniques

  2021cites this paper
• Detection of Turkish Sign Language Using Deep Learning and Image Processing Methods

  2021cites this paper
• Gaussian process regression-based learning rate optimization in convolutional neural networks for medical images classification

  2021cites this paper
• Accurate prediction and further dissection of neonicotinoid elimination in the water treatment by CTS@AgBC using multihead attention-based convolutional neural network combined with the time-dependent Cox regression model.

  2021cites this paper
• Utilization of multilayer perceptron for determining the inverse kinematics of an industrial robotic manipulator

  2021cites this paper
• Automatic Printed Fabric Defect Detection Based on Image Classification Using Modified VGG Network

  2021cites this paper
• Artificial neural network prediction of same-day discharge following primary total knee arthroplasty based on preoperative and intraoperative variables.

  2021cites this paper
• Comparative Design Analysis of Optimized Learning Rate for Convolutional Neural Network

  2021cites this paper
• Automatic defect detection for fabric printing using a deep convolutional neural network

  2021cites this paper
• Multi‐objective Deng's grey incidence analysis, orthogonal optimization, and artificial neural network modelling in hot‐maceration‐assisted extraction of African cucumber leaves (Momordica balsamina)

  2021cites this paper
• Analysis of ANN-based daily global horizontal irradiance prediction models with different meteorological parameters: a case study of mountainous region of India

  2021cites this paper
• A new deep auto-encoder using multiscale reconstruction errors and weight update correlation

  2021cites this paper
• Artificial neural network application in predicting probabilistic seismic demands of bridge components

  2021cites this paper
• A training algorithm with selectable search direction for complex-valued feedforward neural networks

  2021cites this paper
• Tandem solar cells efficiency prediction and optimization via deep learning.

  2021cites this paper
• Artificial classification of cervical squamous lesions in ThinPrep cytologic tests using a deep convolutional neural network

  2020cites this paper
• Real-time estimation of COVID-19 cases using machine learning and mathematical models - The case of India

  2020cites this paper
• Dynamic batch size tuning based on stopping criterion for neural network training

  2020cites this paper
• Crack detection in carbide anvil using acoustic signal and deep learning with particle swarm optimisation

  2020cites this paper
• Vector-Based Back Propagation Algorithm of Supervised Convolution Neural Network

  2020cites this paper