On the importance of initialization and momentum in deep learning

No abstract is available for this paper.

• Publication year

  2013

• Venue

  International Conference on Machine Learning

• Publication date

  2013-06-16

• Fields of study

  Mathematics, Computer Science

• Identifiers

  No identifiers available.

• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Introductory Lectures on Convex Optimization - A Basic Course

  2014cited by this paper
• Deep Learning Made Easier by Linear Transformations in Perceptrons

  2012influential reference
• Acoustic Modeling Using Deep Belief Networks

  2012cited by this paper
• Deep Neural Networks for Acoustic Modeling in Speech Recognition

  2012cited by this paper
• Training Deep and Recurrent Networks with Hessian-Free Optimization

  2012influential reference
• Neural Networks: Tricks of the Trade

  2012cited by this paper
• Context-Dependent Pre-Trained Deep Neural Networks for Large-Vocabulary Speech Recognition

  2012cited by this paper
• ImageNet classification with deep convolutional neural networks

  2012cited by this paper
• Sequence Transduction with Recurrent Neural Networks

  2012influential reference
• Efficient BackProp

  2012cited by this paper
• Learning Recurrent Neural Networks with Hessian-Free Optimization

  2011influential reference
• SUBWORD LANGUAGE MODELING WITH NEURAL NETWORKS

  2011cited by this paper
• Generating Text with Recurrent Neural Networks

  2011influential reference
• Better Mini-Batch Algorithms via Accelerated Gradient Methods

  2011cited by this paper
• Improved Preconditioner for Hessian Free Optimization

  2011cited by this paper
• An optimal method for stochastic composite optimization

  2011influential reference
• Understanding the difficulty of training deep feedforward neural networks

  2010cited by this paper
• Deep learning via Hessian-free optimization

  2010influential reference
• Supporting Online Material for Reducing the Dimensionality of Data with Neural Networks

  2006influential reference
• Greedy Layer-Wise Training of Deep Networks

  2006cited by this paper
• A Fast Learning Algorithm for Deep Belief Nets

  2006influential reference
• Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communication

  2004cited by this paper
• Introductory Lectures on Convex Optimization

  2003cited by this paper
• Large Scale Online Learning

  2003cited by this paper
• Neural Networks: Tricks of the Trade

  2002cited by this paper
• Long Short-Term Memory

  1997cited by this paper
• Dynamics and algorithms for stochastic search

  1996cited by this paper
• Learning long-term dependencies with gradient descent is difficult

  1994influential reference
• Stochastic dynamics of learning with momentum in neural networks

  1994cited by this paper
• Towards Faster Stochastic Gradient Search

  1991cited by this paper
• Some methods of speeding up the convergence of iteration methods

  1964influential reference
• Journal of Graph Algorithms and Applications Many-to-one Boundary Labeling

  year unknowncited by this paper

• Bregman proximal gradient with extrapolation to train a reservoir computing network for a binary classification task

  2026cites this paper
• Essentially No Energy Barrier Between Independent Fermionic Neural Quantum State Minima

  2026cites this paper
• From Injection to Detection: Analyzing the Impact of Anomalous Data in Federated Learning

  2026cites this paper
• Generalized Stochastic Gradient Descent with Momentum Methods for Smooth Optimization

  2026cites this paper
• Improved Convergence Rates of Muon Optimizer for Nonconvex Optimization

  2026cites this paper
• Insights into the behavior and performance of Linear Structured Filter Coefficients (LSFC) in solving one-dimensional inverse heat conduction problems

  2026cites this paper
• Progressive Power Homotopy for Non-convex Optimization

  2026cites this paper
• Continuized Nesterov Momentum Achieves the $O(\varepsilon^{-7/4})$ Complexity without Additional Mechanisms

  2026cites this paper
• Celo2: Towards Learned Optimization Free Lunch

  2026cites this paper
• A novel stochastic conjugate gradient algorithm based on a stochastic differential equation perspective

  2026cites this paper
• Convex Dominance in Deep Learning I: A Scaling Law of Loss and Learning Rate

  2026cites this paper
• On the Convergence of HalpernSGD

  2026cites this paper
• Discriminative response pruning for robust and efficient deep networks under label noise

  2026cites this paper
• S-Leon: An Efficient Split Learning Framework Over Heterogeneous LEO Satellite Networks

  2026cites this paper
• AI-driven exploration and design of inorganic battery materials

  2026cites this paper
• Stabilizing autoregressive forecasts in chaotic systems via multi-rate latent recurrence

  2026cites this paper
• The Double-Edged Sword of Knowledge Transfer: Diagnosing and Curing Fairness Pathologies in Cross-Domain Recommendation

  2026cites this paper
• REDistill: Robust Estimator Distillation for Balancing Robustness and Efficiency

  2026influential citation
• Harnessing the power of Squidle+ to develop flexible machine learning models.

  2026cites this paper
• Ensemble Prediction of Task Affinity for Efficient Multi-Task Learning

  2026cites this paper
• Towards lightweight noise-robust models via noise-aware pruning

  2026influential citation
• Optimization Algorithms for Brain MRI-Based Alzheimer’s Disease Classification: A Comprehensive Review and Methodological Framework

  2026cites this paper
• Fast Compute for ML Optimization

  2026cites this paper
• Acceleration for Polyak-{\L}ojasiewicz Functions with a Gradient Aiming Condition

  2026cites this paper
• Adaptive Momentum and Nonlinear Damping for Neural Network Training

  2026influential citation
• PPI-SVRG: Unifying Prediction-Powered Inference and Variance Reduction for Semi-Supervised Optimization

  2026cites this paper
• Unit-Consistent (UC) Adjoint for GSD and Backprop in Deep Learning Applications

  2026cites this paper
• UAV Cross-Domain Detection in IoT-Enabled Environments via Class-Aware Correlation-Controlled Feature Learning

  2026influential citation
• Probabilistic adaptive learning for enhanced speech emotion recognition in the presence of noisy labels

  2026cites this paper
• Learn from the best: A universal self-distillation approach with historical logits

  2026cites this paper
• PrivESD: A Privacy-Preserving Cloud-Edge Collaborative Logistic Regression Model Over Encrypted Streaming Data

  2026cites this paper
• Multi-Scale Content-Aware Enhancement Dual-Branch CNN for LED-Chip Defect Classification

  2026influential citation
• A Unified Frequency Principle for Quantum and Classical Machine Learning

  2026cites this paper
• Online Continual Learning for Time Series: a Natural Score-driven Approach

  2026influential citation
• Stability and Generalization of Nonconvex Optimization with Heavy-Tailed Noise

  2026cites this paper
• Training Memory in Deep Neural Networks: Mechanisms, Evidence, and Measurement Gaps

  2026influential citation
• Riemannian Lyapunov Optimizer: A Unified Framework for Optimization

  2026cites this paper
• RAWDet-7: A Multi-Scenario Benchmark for Object Detection and Description on Quantized RAW Images

  2026cites this paper
• Feature adaptive extraction and gated fusion networks for road crack segmentation

  2026cites this paper
• Multi-scale metric learning neural network with adaptive graphs for few-shot gas path fault diagnosis of variable cycle engine

  2026cites this paper
• SecureGate: Learning When to Reveal PII Safely via Token-Gated Dual-Adapters for Federated LLMs

  2026cites this paper
• D33D-net: A robust dynamic deformable dual-pathway 3D deep neural network with embedded handcrafted features for liver tumor segmentation in CT images

  2026cites this paper
• CRISPR-DEO: Decision-Aware Economic Dispatch Optimization via Sparse Gradient Editing for Power System Forecasting

  2026cites this paper
• Segmenting Low-Contrast XCTs of Concretes: An Unsupervised Approach

  2026cites this paper
• A Context-Free Model of Savings in Motor Learning

  2026cites this paper
• Multi-Prototypes Representation Learning for Contrastive Clustering

  2026cites this paper
• Momentum Guidance: Plug-and-Play Guidance for Flow Models

  2026cites this paper
• Feed forward neural network for non-intrusive load monitoring

  2026cites this paper
• FGRPO: Fine-grained reasoning preference optimization with iterative stabilization for mathematical reasoning in large language model

  2026cites this paper
• Musical Metamerism with Time--Frequency Scattering

  2026cites this paper
• Carbon–phosphorus exchange rate constrains density–speed trade-off in arbuscular mycorrhizal fungal growth

  2026cites this paper
• Enhanced accuracy through ensembling of randomly initialized auto-regressive models for dynamical systems

  2026cites this paper
• Stein-Rule Shrinkage for Stochastic Gradient Estimation in High Dimensions

  2026cites this paper
• Why Adam Works Better with β1 = β2: The Missing Gradient Scale Invariance Principle

  2026cites this paper
• Process-Tensor Tomography of SGD: Measuring Non-Markovian Memory via Back-Flow of Distinguishability

  2026cites this paper
• Controlling Long-Horizon Behavior in Language Model Agents with Explicit State Dynamics

  2026cites this paper
• On the use of deep learning for computer-generated holography

  2025cites this paper
• Preconditioning Natural and Second Order Gradient Descent in Quantum Optimization: A Performance Benchmark

  2025cites this paper
• AlphaGrad: Non-Linear Gradient Normalization Optimizer

  2025cites this paper
• Seeking Flat Minima over Diverse Surrogates for Improved Adversarial Transferability: A Theoretical Framework and Algorithmic Instantiation

  2025cites this paper
• Adaptive adam-based optimizers using second-order weight decoupling and gradient-aware weight decay for vision transformer

  2025cites this paper
• PACE: A Framework for Learning and Control in Linear Incomplete-Information Differential Games

  2025cites this paper
• Enhancing Fire Detection with YOLO Models: A Bayesian Hyperparameter Tuning Approach

  2025cites this paper
• A contrast enhanced representation normalization approach to knowledge distillation

  2025cites this paper
• Enhancing skin lesion classification: a CNN approach with human baseline comparison

  2025cites this paper
• AdaptoVision: A Multi-Resolution Image Recognition Model for Robust and Scalable Classification

  2025cites this paper
• A Lightweight Network Intrusion Detection System Based on Temporal Convolutional Networks and Attention Mechanisms

  2025cites this paper
• Arnet: research on wood CT image classification algorithm based on multi-scale dilated attention and residual dynamic convolution

  2025influential citation
• Effects of Momentum in Implicit Bias of Gradient Flow for Diagonal Linear Networks

  2025cites this paper
• CONTINA: Confidence Interval for Traffic Demand Prediction with Coverage Guarantee

  2025cites this paper
• Financial Data Analysis with Robust Federated Logistic Regression

  2025cites this paper
• Proximal recursive generalized hyper-gradient descent method

  2025cites this paper
• Analyzing and Optimizing Perturbation of DP-SGD Geometrically

  2025cites this paper
• Extended multiphysics-informed neural network for conjugate heat transfer problems

  2025cites this paper
• Exploiting Gaussian distribution in channel pruning for convolutional neural networks

  2025influential citation
• Neighborhood relation-based knowledge distillation for image classification

  2025cites this paper
• A Multi-Modal AI System for Screening Mammography: Integrating 2D and 3D Imaging to Improve Breast Cancer Detection in a Prospective Clinical Study

  2025cites this paper
• Foundation model of neural activity predicts response to new stimulus types

  2025cites this paper
• Reservoir Computing: A New Paradigm for Neural Networks

  2025cites this paper
• Learning from Streaming Video with Orthogonal Gradients

  2025cites this paper
• Comparative Analysis of Gradient-Based Optimizers in Feedforward Neural Networks for Titanic Survival Prediction

  2025cites this paper
• Prediction and Prioritisation of Novel Anthelmintic Candidates from Public Databases Using Deep Learning and Available Bioactivity Data Sets

  2025influential citation
• Optimization Techniques in Machine Learning: A Comprehensive Review

  2025cites this paper
• Optimization of Photovoltaic Power Forecasting: A Comparative Study of Deep Learning Architectures, Optimization Techniques, and Evaluation Metrics

  2025cites this paper
• Multi-Pathway 3D CNN With Conditional Random Field for Automated Segmentation of Multiple Sclerosis Lesions in MRI

  2025cites this paper
• HVAdam: A Full-Dimension Adaptive Optimizer

  2025influential citation
• Combinatorial optimization: From deep learning to large language models

  2025cites this paper
• Pseudo-Asynchronous Local SGD: Robust and Efficient Data-Parallel Training

  2025cites this paper
• Explainable Bayesian deep learning through input-skip Latent Binary Bayesian Neural Networks

  2025cites this paper
• Machine learning for single-ended event reconstruction in PROSPECT experiment

  2025cites this paper
• Finding the Muses: Identifying Coresets through Loss Trajectories

  2025cites this paper
• Training large-scale language models with limited GPU memory: a survey

  2025cites this paper
• Uncertainty-aware Long-tailed Weights Model the Utility of Pseudo-labels for Semi-supervised Learning

  2025influential citation
• Communication-Efficient Language Model Training Scales Reliably and Robustly: Scaling Laws for DiLoCo

  2025influential citation
• A Conflict-Guided Evidential Multimodal Fusion for Semantic Segmentation

  2025cites this paper
• Intra-class progressive and adaptive self-distillation

  2025cites this paper
• Improving generalization in DNNs through enhanced orthogonality in momentum-based optimizers

  2025cites this paper
• Instance-Level Data-Use Auditing of Visual ML Models

  2025cites this paper
• Momentum-Based Iterative Hard Thresholding Algorithm for Sparse Signal Recovery

  2025cites this paper
• Generalization abilities of foundation models in waste classification.

  2025influential citation