Low-Dimensional Adaptive Neural Network Regression With Directional Change Detection via Nuclear Norm Penalization

This paper reports on our study of a regularized neural network regression method that adapts to the low‐dimensional directional structure of a function. The key innovation lies in identifying direction vectors where the function exhibits significant variation and conducting estimation within this reduced‐dimensional space. This is achieved through a regularization scheme that controls the nuclear norm of the weight matrix. The nuclear norm penalization approach effectively reduces its rank and allows the model to discover the principal subspace where the function varies. Additionally, an penalty is imposed to introduce node‐level sparsity, which further enhances dimension reduction and improves estimation efficiency. The combination of the nuclear norm and norm penalties results in a low‐dimensional network structure that strikes a good balance in the bias‐variance trade‐off and improves estimator performance. An efficient and stable implementation scheme is designed based on the alternating direction method of multipliers and the Levenberg–Marquardt algorithm. The stability of the algorithm is enhanced by using a B‐spline activation function with compact support and an initialization strategy based on model‐based sliced inverse regression. Numerical experiments on simulated and benchmark datasets demonstrate that our method outperforms several popular machine learning and neural network regression techniques.

• Publication year

  2025

• Venue

  Statistical analysis and data mining

• Publication date

  2025-07-26

• Fields of study

  Computer Science

• Identifiers
  DOI 10.1002/sam.70035
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Online dimensionality reduction through stacked generalization of spectral methods with deep networks

  2025cited by this paper
• An Adaptive Neural Network Regression Method for Structure Identification

  2023cited by this paper
• Sparse neural network regression with variable selection

  2022cited by this paper
• Minimax estimation in multi-task regression under low-rank structures

  2022cited by this paper
• Multivariate Reduced-Rank Regression

  2022cited by this paper
• LassoNet: A Neural Network with Feature Sparsity

  2019cited by this paper
• Adaptive Approximation and Generalization of Deep Neural Network with Intrinsic Dimensionality

  2019cited by this paper
• Sufficient Dimension Reduction: Methods and Applications with R

  2018cited by this paper
• XGBoost: A Scalable Tree Boosting System

  2016cited by this paper
• Group sparse regularization for deep neural networks

  2016cited by this paper
• Minimax risk of matrix denoising by singular value thresholding

  2013cited by this paper
• Model-based SIR for dimension reduction

  2011cited by this paper
• A Singular Value Thresholding Algorithm for Matrix Completion

  2008cited by this paper
• Dimension reduction and coefficient estimation in multivariate linear regression

  2007cited by this paper
• Supporting Online Material for Reducing the Dimensionality of Data with Neural Networks

  2006cited by this paper
• Component selection and smoothing in multivariate nonparametric regression

  2006cited by this paper
• Adaptive regression estimation with multilayer feedforward neural networks

  2004cited by this paper
• The Elements of Statistical Learning: Data Mining, Inference, and Prediction

  2001cited by this paper
• The Use of Polynomial Splines and Their Tensor Products in Multivariate Function Estimation

  1994cited by this paper
• A general regression neural network

  1991cited by this paper
• Multivariate adaptive regression splines

  1990cited by this paper
• A Lower Bound on the Error in Nonparametric Regression Type Problems

  1988cited by this paper
• Generalized Additive Models: Some Applications

  1987cited by this paper
• The Dimensionality Reduction Principle for Generalized Additive Models

  1986cited by this paper
• Additive Regression and Other Nonparametric Models

  1985cited by this paper
• Neural networks and physical systems with emergent collective computational abilities.

  1982cited by this paper
• Optimal Global Rates of Convergence for Nonparametric Regression

  1982cited by this paper
• Estimating the Dimension of a Model

  1978cited by this paper
• A Practical Guide to Splines

  1978cited by this paper

• No citing papers are available for this paper.