Three ways that non-differentiability affects neural network training

No abstract is available for this paper.

• Publication year

  2024

• Venue

  arXiv.org

• Publication date

  Unknown publication date

• Fields of study

  Mathematics, Computer Science

• Identifiers
  DOI 10.48550/arXiv.2401.08426
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Analytical solutions for a family of single layer neural network regression problems

  2023cited by this paper
• Adam Can Converge Without Any Modification on Update Rules

  2022influential reference
• Learning threshold neurons via the "edge of stability"

  2022cited by this paper
• Understanding the unstable convergence of gradient descent

  2022influential reference
• The Multiscale Structure of Neural Network Loss Functions: The Effect on Optimization and Origin

  2022influential reference
• Understanding Gradient Descent on Edge of Stability in Deep Learning

  2022cited by this paper
• Analyzing Sharpness along GD Trajectory: Progressive Sharpening and Edge of Stability

  2022cited by this paper
• On Gradient Descent Convergence beyond the Edge of Stability

  2022cited by this paper
• Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability

  2021cited by this paper
• Numerical influence of ReLU'(0) on backpropagation

  2021cited by this paper
• Nonlinear Programming

  2021influential reference
• Efficient Prediction of Cardiovascular Disease Using Machine Learning Algorithms With Relief and LASSO Feature Selection Techniques

  2021cited by this paper
• RMSprop converges with proper hyper-parameter

  2021cited by this paper
• Training Sparse Neural Networks using Compressed Sensing

  2020cited by this paper
• A Simple Convergence Proof of Adam and Adagrad

  2020cited by this paper
• Deep Learning With PyTorch

  2020influential reference
• What is the State of Neural Network Pruning?

  2020influential reference
• Conservative set valued fields, automatic differentiation, stochastic gradient methods and deep learning

  2019cited by this paper
• Regularization for Deep Learning

  2019cited by this paper
• Differentiable Convex Optimization Layers

  2019cited by this paper
• Large Batch Optimization for Deep Learning: Training BERT in 76 minutes

  2019cited by this paper
• Variable Selection

  2019cited by this paper
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks

  2018cited by this paper
• Improved Adam Optimizer for Deep Neural Networks

  2018cited by this paper
• On the Convergence of Adam and Beyond

  2018cited by this paper
• Stochastic Subgradient Method Converges on Tame Functions

  2018cited by this paper
• Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data

  2018cited by this paper
• The Multilinear Structure of ReLU Networks

  2017cited by this paper
• OptNet: Differentiable Optimization as a Layer in Neural Networks

  2017cited by this paper
• Automatic differentiation in PyTorch

  2017influential reference
• Pruning Filters for Efficient ConvNets

  2016cited by this paper
• Minimization Methods For Non Differentiable Functions

  2016cited by this paper
• Group sparse regularization for deep neural networks

  2016cited by this paper
• Neural networks with differentiable structure

  2016cited by this paper
• SSD: Single Shot MultiBox Detector

  2015cited by this paper
• Statistical Learning with Sparsity: The Lasso and Generalizations

  2015influential reference
• Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding

  2015cited by this paper
• Fast R-CNN

  2015cited by this paper
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

  2015cited by this paper
• Learning both Weights and Connections for Efficient Neural Network

  2015cited by this paper
• Identifying and attacking the saddle point problem in high-dimensional non-convex optimization

  2014cited by this paper
• Genomic selection using regularized linear regression models: ridge regression, lasso, elastic net and their extensions

  2012cited by this paper
• Practical Recommendations for Gradient-Based Training of Deep Architectures

  2012cited by this paper
• Overview of LASSO-related penalized regression methods for quantitative trait mapping and genomic selection

  2012cited by this paper
• Adaptive Subgradient Methods for Online Learning and Stochastic Optimization

  2011cited by this paper
• The Graphical Lasso: New Insights and Alternatives

  2011cited by this paper
• The Bayesian lasso for genome-wide association studies

  2011cited by this paper
• Convex optimization

  2010cited by this paper
• Large-Scale Machine Learning with Stochastic Gradient Descent

  2010cited by this paper
• Adaptive Bound Optimization for Online Convex Optimization

  2010cited by this paper
• 双対平坦空間におけるLeast Angle Regressionと情報量規準

  2009cited by this paper
• Dual Averaging Methods for Regularized Stochastic Learning and Online Optimization

  2009cited by this paper
• Subgradient Methods

  2007influential reference
• Sparsity and smoothness via the fused lasso

  2005cited by this paper
• Regression Shrinkage and Selection via the Lasso

  1996cited by this paper
• The Subgradient Method

  1985cited by this paper

• No citing papers are available for this paper.