Deep Learning and MARS: A Connection

We consider least squares regression estimates using deep neural networks. We show that these estimates satisfy an oracle inequality, which implies that (up to a logarithmic factor) the error of these estimates is at least as small as the optimal possible error bound which one would expect for MARS in case that this procedure would work in the optimal way. As a result we show that our neural networks are able to achieve a dimensionality reduction in case that the regression function locally has low dimensionality. This assumption seems to be realistic in real-world applications, since selected high-dimensional data are often confined to locally-low-dimensional distributions. In our simulation study we provide numerical experiments to support our theoretical results and to compare our estimate with other conventional nonparametric regression estimates, especially with MARS. The use of our estimates is illustrated through a real data analysis.

• Publication year

  2019

• Venue

  arXiv.org

• Publication date

  2019-08-29

• Fields of study

  Mathematics, Computer Science

• Identifiers
  arXiv 1908.11140
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Multivariate Adaptive Regression Splines

  2019influential reference
• On deep learning as a remedy for the curse of dimensionality in nonparametric regression

  2019influential reference
• A comparison of deep networks with ReLU activation function and linear spline-type methods

  2018influential reference
• Deep Neural Networks Learn Non-Smooth Functions Effectively

  2018cited by this paper
• Nonparametric Regression Based on Hierarchical Interaction Models

  2017cited by this paper
• Nonparametric regression using deep neural networks with ReLU activation function

  2017influential reference
• Nonparametric estimation of a function from noiseless observations at random points

  2017cited by this paper
• Adaptive Regression Splines toolbox for Matlab/Octave

  2015cited by this paper
• Investigating Smooth Multiple Regression by the Method of Average Derivatives

  2015cited by this paper
• Optimal global rates of convergence for noiseless regression estimation problems with adaptively chosen design

  2014influential reference
• Event labeling combining ensemble detectors and background knowledge

  2014cited by this paper
• Deep learning in neural networks: An overview

  2014cited by this paper
• Adaptive estimation under single-index constraint in a regression model

  2013cited by this paper
• Estimation of a Regression Function by Maxima of Minima of Linear Functions

  2009cited by this paper
• Local Dimensionality Reduction for Non-Parametric Regression

  2009cited by this paper
• Rate-optimal estimation for a general class of nonparametric regression models with unknown link functions

  2007cited by this paper
• Variable selection for the single‐index model

  2007cited by this paper
• Adaptive regression estimation with multilayer feedforward neural networks

  2004cited by this paper
• Radial basis functions for the multivariate interpolation of large scattered data sets

  2002cited by this paper
• Penalized Spline Estimation for Partially Linear Single-Index Models

  2002cited by this paper
• A Distribution-Free Theory of Nonparametric Regression

  2002influential reference
• A Distribution-Free Theory of Nonparametric Regression (Springer Series in Statistics)

  2002influential reference
• Universal Approximation Using Feedforward Neural Networks: A Survey of Some Existing Methods, and Some New Results

  1998cited by this paper
• Local Dimensionality Reduction

  1997cited by this paper
• Local dimensionality reduction for locally weighted learning

  1997cited by this paper
• A Probabilistic Theory of Pattern Recognition

  1996influential reference
• Convergence rates for single hidden layer feedforward networks

  1994cited by this paper
• The Use of Polynomial Splines and Their Tensor Products in Multivariate Function Estimation

  1994cited by this paper
• Universal approximation bounds for superpositions of a sigmoidal function

  1993cited by this paper
• Optimal Smoothing in Single-index Models

  1993cited by this paper
• Approximation and Estimation Bounds for Artificial Neural Networks

  1991cited by this paper
• What is projection pursuit

  1987cited by this paper
• Additive Regression and Other Nonparametric Models

  1985cited by this paper
• Optimal Global Rates of Convergence for Nonparametric Regression

  1982cited by this paper
• Projection Pursuit Regression

  1981cited by this paper
• Distribution-Free Consistency Results in Nonparametric Discrimination and Regression Function Estimation

  1980influential reference

• On the rate of convergence of image classifiers based on convolutional neural networks

  2020cites this paper
• Approximating smooth functions by deep neural networks with sigmoid activation function

  2020cites this paper
• On the rate of convergence of image classi(cid:28)ers based on convolutional neural networks 1

  2020cites this paper
• Discussion of: “Nonparametric regression using deep neural networks with ReLU activation function”

  2020cites this paper
• Over-parametrized deep neural networks do not generalize well

  2019cites this paper
• Analysis of the rate of convergence of neural network regression estimates which are easy to implement.

  2019cites this paper
• On the rate of convergence of a neural network regression estimate learned by gradient descent

  2019cites this paper