Forward stagewise regression and the monotone lasso

We consider the least angle regression and forward stagewise algorithms for solving penalized least squares regression problems. In Efron, Hastie, Johnstone & Tibshirani (2004) it is proved that the least angle regression algorithm, with a small modification, solves the lasso regression problem. Here we give an analogous result for incremental forward stagewise regression, showing that it solves a version of the lasso problem that enforces monotonicity. One consequence of this is as follows: while lasso makes optimal progress in terms of reducing the residual sum-of-squares per unit increase in $L_1$-norm of the coefficient $\beta$, forward stage-wise is optimal per unit $L_1$ arc-length traveled along the coefficient path. We also study a condition under which the coefficient paths of the lasso are monotone, and hence the different algorithms coincide. Finally, we compare the lasso and forward stagewise procedures in a simulation study involving a large number of correlated predictors.

• Publication year

  2007

• Venue

  arXiv: Statistics Theory

• Publication date

  2007-05-02

• Fields of study

  Mathematics

• Identifiers
  DOI 10.1214/07-EJS004arXiv 0705.0269
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• 双対平坦空間におけるLeast Angle Regressionと情報量規準

  2009cited by this paper
• Fast Solution of $\ell _{1}$ -Norm Minimization Problems When the Solution May Be Sparse

  2008cited by this paper
• Piecewise linear regularized solution paths

  2007cited by this paper
• L1‐regularization path algorithm for generalized linear models

  2007cited by this paper
• Boosting for high-dimensional linear models

  2006cited by this paper
• Annals of Statistics

  2006influential reference
• Just relax: convex programming methods for identifying sparse signals in noise

  2006cited by this paper
• Gradient Directed Regularization

  2004cited by this paper
• Greed is good: algorithmic results for sparse approximation

  2004cited by this paper
• Boosted Lasso

  2004cited by this paper
• Tracking Curved Regularized Optimization Solution Paths

  2004cited by this paper
• The Elements of Statistical Learning: Data Mining, Inference, and Prediction

  2004influential reference
• Greedy function approximation: A gradient boosting machine.

  2001cited by this paper
• Special Invited Paper-Additive logistic regression: A statistical view of boosting

  2000cited by this paper
• A new approach to variable selection in least squares problems

  2000cited by this paper
• Atomic Decomposition by Basis Pursuit

  1998cited by this paper
• Boosting the margin: A new explanation for the effectiveness of voting methods

  1997cited by this paper
• Regression Shrinkage and Selection via the Lasso

  1996influential reference

• New explanations and inference for least angle regression

  2026cites this paper
• Analysis of the Travelling Time According to Weather Conditions Using Machine Learning Algorithms

  2025cites this paper
• Diagonal Linear Networks and the Lasso Regularization Path

  2025cites this paper
• Fitting high-dimensional mixture cure models using the hdcuremodels R package

  2025cites this paper
• Gradient Boosting for Spatial Regression Models with Autoregressive Disturbances

  2025cites this paper
• Gradient boosting for Dirichlet regression models

  2025cites this paper
• Interpreting LASSO regression model by feature space matching analysis for spatio-temporal correlation based wind power forecasting

  2025cites this paper
• Foundations and Innovations in Data Fusion and Ensemble Learning for Effective Consensus

  2025cites this paper
• Measures of Uncertainty for Shrinkage Model Selection

  2024cites this paper
• Ensemble with Divisive Bagging for Feature Selection in Big Data

  2024cites this paper
• Sparse Regression for Machine Translation

  2024influential citation
• Enhanced prediction of cement raw meal oxides by near-infrared spectroscopy using machine learning combined with chemometric techniques

  2024cites this paper
• A Weibull mixture cure frailty model for high-dimensional covariates

  2024influential citation
• Solving Kernel Ridge Regression with Gradient-Based Optimization Methods

  2023cites this paper
• Forward stagewise regression with multilevel memristor for sparse coding

  2023cites this paper
• Fast robust kernel regression through sign gradient descent with early stopping

  2023influential citation
• Eelbrain, a Python toolkit for time-continuous analysis with temporal response functions

  2023cites this paper
• Optimal Sets and Solution Paths of ReLU Networks

  2023cites this paper
• E LASTIC G RADIENT D ESCENT AND E LASTIC G RADIENT F LOW : LARS L IKE A LGORITHMS A PPROXIMATING THE S OLUTION P ATHS OF THE E LASTIC N ET

  2022cites this paper
• Efficient least angle regression for identification of linear-in-the-parameters models

  2022cites this paper
• The Solution Path of the Group Lasso

  2022cites this paper
• Elastic Gradient Descent, an Iterative Optimization Method Approximating the Solution Paths of the Elastic Net

  2022cites this paper
• Parameter expansion for fitting regression models with non-negativity constraints

  2022cites this paper
• Decomposable Sparse Tensor on Tensor Regression

  2022cites this paper
• Discovering Reduced-Order Dynamical Models From Data Discovering Reduced-Order Dynamical Models From Data

  2022cites this paper
• Controlled variable selection in Weibull mixture cure models for high‐dimensional data

  2022influential citation
• Adaptive step-length selection in gradient boosting for Gaussian location and scale models

  2022cites this paper
• Predicting Fuel Consumption in Power Generation Plants using Machine Learning and Neural Networks

  2021cites this paper
• Randomized boosting with multivariable base-learners for high-dimensional variable selection and prediction

  2021cites this paper
• Data for: Eelbrain: A Python toolkit for time-continuous analysis with temporal response functions

  2021cites this paper
• Eelbrain: A Python toolkit for time-continuous analysis with temporal response functions

  2021cites this paper
• Kernel-based L_2-Boosting with Structure Constraints

  2020cites this paper
• Improving symbolic regression based on correlation between residuals and variables

  2020cites this paper
• Efficient Super Greedy Boosting for Classification

  2020cites this paper
• Solution of Linear Systems

  2020cites this paper
• Fully-Corrective Gradient Boosting with Squared Hinge: Fast Learning Rates and Early Stopping

  2020cites this paper
• Learning outside the Black-Box: The pursuit of interpretable models

  2020cites this paper
• A sparse linear regression model for incomplete datasets

  2019influential citation
• Rescaled Boosting in Classification

  2019cites this paper
• Forward-stagewise clustering: An algorithm for convex clustering

  2019cites this paper
• A tutorial on fitting discrete response models in high-dimensional datasets with the countgmifs package

  2019cites this paper
• A Feature Selection Framework for Software Defect Prediction Using ISFLA

  2019cites this paper
• Developments in Psychometric Population Models for Technology-Based Large-Scale Assessments: An Overview of Challenges and Opportunities

  2019cites this paper
• An Exploratory Analysis of Biased Learners in Soft-Sensing Frames

  2019cites this paper
• Statistical guarantees for local graph clustering

  2019cites this paper
• Uplift Regression: The R Package tools4uplift

  2019cites this paper
• Boosting High Dimensional Predictive Regressions with Time Varying Parameters

  2019influential citation
• On the Differences between L2-Boosting and the Lasso

  2018cites this paper
• A Forward and Backward Stagewise algorithm for nonconvex loss functions with adaptive Lasso

  2018cites this paper
• Nonnegative OPLS for Supervised Design of Filter Banks: Application to Image and Audio Feature Extraction

  2018cites this paper
• Distributed Multivariate Regression Modeling For Selecting Biomarkers Under Data Protection Constraints

  2018cites this paper
• Guide on performing feature selection with the R package MXM

  2018cites this paper
• Prediction of internal egg quality characteristics and variable selection using regularization methods: ridge, LASSO and elastic net

  2018cites this paper
• Reduced order surrogate modeling technique for linear dynamic systems

  2018cites this paper
• Polynomial chaos expansions and stochastic nite-element methods

  2018cites this paper
• Penalized likelihood methods for covariance selection in the context of non-stationary data

  2017cites this paper
• LLR: a latent low‐rank approach to colocalizing genetic risk variants in multiple GWAS

  2017cites this paper
• An Update on Statistical Boosting in Biomedicine

  2017cites this paper
• Efficient least angle regression for identification of linear-in-the-parameters models

  2017cites this paper
• Probabilistic Rule Realization and Selection

  2017cites this paper
• A framework for adaptive regularization in streaming Lasso models

  2016cites this paper
• INFORMS OS Today

  2016cites this paper
• Selecting Spatial Scale of Area-Level Covariates in Regression Models

  2016influential citation
• An extension of penalized ordinal response models

  2016influential citation
• Approaches to Regularized Regression – A Comparison between Gradient Boosting and the Lasso

  2016influential citation
• Nonlinear variable selection with continuous outcome: A fully nonparametric incremental forward stagewise approach

  2016influential citation
• Sparse polynomial chaos expansions of frequency response functions using stochastic frequency transformation

  2016cites this paper
• Causal Compression

  2016cites this paper
• Algorithms for Fitting the Constrained Lasso

  2016cites this paper
• Modeling Discrete Survival Time Using Genomic Feature Data

  2015cites this paper
• Design and Analysis of Clinical Trials for Predictive Medicine

  2015cites this paper
• Polynomial chaos expansions and stochastic finite element methods

  2015cites this paper
• Some Improvements and Applications of Non-intrusive Polynomial Chaos Expansions

  2015cites this paper
• Chaospy: An open source tool for designing methods of uncertainty quantification

  2015cites this paper
• Classification of team sport activities using a single wearable tracking device.

  2015cites this paper
• Re-scale boosting for regression and classification

  2015cites this paper
• Shrinkage Degree in $L_{2}$ -Rescale Boosting for Regression

  2015influential citation
• High-Throughput Data Analysis: Application to Micronuclei Frequency and T-cell Receptor Sequencing

  2015influential citation
• Learning Dynamic Feature Selection for Fast Sequential Prediction

  2015cites this paper
• Hyperspectral Image Classification by Nonlocal Joint Collaborative Representation With a Locally Adaptive Dictionary

  2014cites this paper
• ordinalgmifs: An R Package for Ordinal Regression in High-dimensional Data Settings

  2014cites this paper
• Learning Methods for Variable Selection and Time Series Prediction

  2014cites this paper
• Awards and Honors

  2014cites this paper
• Efficient Bayesian inference of subsurface flow models using nested sampling and sparse polynomial chaos surrogates

  2014cites this paper
• A tutorial on fitting ordinal response models in high-dimensional datasets with the ordinalgmifs package

  2014cites this paper
• A Survey of L1 Regression

  2013cites this paper
• Sparse Solution of Fiber Orientation Distribution Function by Diffusion Decomposition

  2013cites this paper
• Tailoring sparse multivariable regression techniques for prognostic single‐nucleotide polymorphism signatures

  2013cites this paper
• PUMA: A Unified Framework for Penalized Multiple Regression Analysis of GWAS Data

  2013cites this paper
• Modeling Biological Processes In Genome-Wide Association Studies Using Regularized Regression

  2013cites this paper
• AdaBoost and Forward Stagewise Regression are First-Order Convex Optimization Methods

  2013influential citation
• Fine-Mapping the Genetic Association of the Major Histocompatibility Complex in Multiple Sclerosis: HLA and Non-HLA Effects

  2013cites this paper
• On the Semisupervised Joint Trained Elastic Net

  2013cites this paper
• Study on the ensemble methods with kernel ridge regression

  2012cites this paper
• L1-based compression of random forest models

  2012cites this paper
• Which environmental variables should I use in my biodiversity model?

  2012cites this paper
• Keeping greed good: sparse regression under design uncertainty with application to biomass characterization

  2012cites this paper
• CHARACTERIZING L 2 BOOSTING

  2012cites this paper
• A Model Selection Method for Nonlinear System Identification Based fMRI Effective Connectivity Analysis

  2011cites this paper
• RegMT System for Machine Translation, System Combination, and Evaluation

  2011cites this paper