Deep Linear Networks Dynamics: Low-Rank Biases Induced by Initialization Scale and L2 Regularization

No abstract is available for this paper.

• Publication year

  2021

• Venue

  arXiv.org

• Publication date

  Unknown publication date

• Fields of study

  Mathematics, Computer Science

• Identifiers

  No identifiers available.

• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances

  2021cited by this paper
• Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learning

  2020influential reference
• A Unifying View on Implicit Bias in Training Linear Neural Networks

  2020cited by this paper
• Implicit Bias in Deep Linear Classification: Initialization Scale vs Training Accuracy

  2020cited by this paper
• Kernel Alignment Risk Estimator: Risk Prediction from Training Data

  2020cited by this paper
• Directional convergence and alignment in deep learning

  2020influential reference
• Implicit Regularization in Deep Learning May Not Be Explainable by Norms

  2020cited by this paper
• Toward a theory of optimization for over-parameterized systems of non-linear equations: the lessons of deep learning

  2020influential reference
• Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Loss

  2020influential reference
• Implicit Regularization of Random Feature Models

  2020cited by this paper
• On Exact Computation with an Infinitely Wide Neural Net

  2019cited by this paper
• Wide neural networks of any depth evolve as linear models under gradient descent

  2019cited by this paper
• Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit

  2019cited by this paper
• First-order methods almost always avoid strict saddle points

  2019cited by this paper
• Implicit Regularization of Discrete Gradient Dynamics in Deep Linear Neural Networks

  2019influential reference
• Implicit Regularization in Deep Matrix Factorization

  2019influential reference
• Gradient Descent Maximizes the Margin of Homogeneous Neural Networks

  2019cited by this paper
• Dynamics of Deep Neural Networks and Neural Tangent Hierarchy

  2019cited by this paper
• The asymptotic spectrum of the Hessian of DNN throughout training

  2019cited by this paper
• Kernel and Rich Regimes in Overparametrized Models

  2019cited by this paper
• A Note on Lazy Training in Supervised Differentiable Programming

  2018cited by this paper
• Characterizing Implicit Bias in Terms of Optimization Geometry

  2018influential reference
• Spectral norm of a symmetric tensor and its computation

  2018cited by this paper
• A mean field view of the landscape of two-layer neural networks

  2018cited by this paper
• On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport

  2018cited by this paper
• Implicit Bias of Gradient Descent on Linear Convolutional Networks

  2018influential reference
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks

  2018cited by this paper
• Gradient descent aligns the layers of deep linear networks

  2018influential reference
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks

  2018cited by this paper
• A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks

  2018cited by this paper
• A mathematical theory of semantic development in deep neural networks

  2018cited by this paper
• Parameters as interacting particles: long time convergence and asymptotic error scaling of neural networks

  2018cited by this paper
• High-dimensional dynamics of generalization error in neural networks

  2017cited by this paper
• The Implicit Bias of Gradient Descent on Separable Data

  2017influential reference
• Deep Learning without Poor Local Minima

  2016cited by this paper
• Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Łojasiewicz Condition

  2016influential reference
• Gradient Descent Only Converges to Minimizers

  2016influential reference
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks

  2013cited by this paper
• Minimax risk of matrix denoising by singular value thresholding

  2013cited by this paper
• The phase transition of matrix recovery from Gaussian measurements matches the minimax MSE of matrix denoising

  2013cited by this paper
• Low-Rank Matrix Approximation with Weights or Missing Data Is NP-Hard

  2010cited by this paper
• Neural networks and principal component analysis: Learning from examples without local minima

  1989cited by this paper
• Über homogene Polynome in ($L^{2}$)

  1938influential reference

• A Generalization Bound for Nearly-Linear Networks

  2024cites this paper
• Understanding Incremental Learning of Gradient Descent: A Fine-grained Analysis of Matrix Sensing

  2023cites this paper
• The Inductive Bias of Flatness Regularization for Deep Matrix Factorization

  2023cites this paper
• Implicit Bias of Gradient Descent on Reparametrized Models: On Equivalence to Mirror Descent

  2022cites this paper
• Robust Training under Label Noise by Over-parameterization

  2022cites this paper
• Self-consistent dynamical field theory of kernel evolution in wide neural networks

  2022cites this paper
• On the Implicit Bias in Deep-Learning Algorithms

  2022cites this paper
• A Dynamics Theory of Implicit Regularization in Deep Low-Rank Matrix Factorization

  2022cites this paper
• Neural Networks as Kernel Learners: The Silent Alignment Effect

  2021cites this paper
• Variational autoencoders in the presence of low-dimensional data: landscape and implicit bias

  2021cites this paper
• Adaptive and Implicit Regularization for Matrix Completion

  2021cites this paper
• Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networks

  2021cites this paper