Extended Comparisons of Best Subset Selection, Forward Stepwise Selection, and the Lasso

In exciting new work, Bertsimas et al. (2016) showed that the classical best subset selection problem in regression modeling can be formulated as a mixed integer optimization (MIO) problem. Using recent advances in MIO algorithms, they demonstrated that best subset selection can now be solved at much larger problem sizes that what was thought possible in the statistics community. They presented empirical comparisons of best subset selection with other popular variable selection procedures, in particular, the lasso and forward stepwise selection. Surprisingly (to us), their simulations suggested that best subset selection consistently outperformed both methods in terms of prediction accuracy. Here we present an expanded set of simulations to shed more light on these comparisons. The summary is roughly as follows: (a) neither best subset selection nor the lasso uniformly dominate the other, with best subset selection generally performing better in high signal-to-noise (SNR) ratio regimes, and the lasso better in low SNR regimes; (b) best subset selection and forward stepwise perform quite similarly throughout; (c) the relaxed lasso (actually, a simplified version of the original relaxed estimator defined in Meinshausen, 2007) is the overall winner, performing just about as well as the lasso in low SNR scenarios, and as well as best subset selection in high SNR scenarios.

• Publication year

  2017

• Venue

  arXiv: Methodology

• Publication date

  2017-07-27

• Fields of study

  Mathematics, Computer Science

• Identifiers
  arXiv 1707.08692
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Effective degrees of freedom : a flawed metaphor

  2015cited by this paper
• Best Subset Selection via a Modern Optimization Lens

  2015influential reference
• When does more regularization imply fewer degrees of freedom? Sufficient conditions and counterexamples

  2014cited by this paper
• The Lasso Problem and Uniqueness

  2012cited by this paper
• Degrees of freedom in lasso problems

  2011cited by this paper
• SparseNet: Coordinate Descent With Nonconvex Penalties

  2011influential reference
• Improved variable selection with Forward-Lasso adaptive shrinkage

  2011cited by this paper
• Regularization Paths for Generalized Linear Models via Coordinate Descent.

  2010cited by this paper
• Strong rules for discarding predictors in lasso‐type problems

  2010cited by this paper
• Square-Root Lasso: Pivotal Recovery of Sparse Signals via Conic Programming

  2010influential reference
• Nearly unbiased variable selection under minimax concave penalty

  2010cited by this paper
• 双対平坦空間におけるLeast Angle Regressionと情報量規準

  2009cited by this paper
• Relaxed Lasso

  2007influential reference
• PATHWISE COORDINATE OPTIMIZATION

  2007cited by this paper
• On the “degrees of freedom” of the lasso

  2007cited by this paper
• The Dantzig selector: Statistical estimation when P is much larger than n

  2005cited by this paper
• Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties

  2001cited by this paper
• The Elements of Statistical Learning: Data Mining, Inference, and Prediction

  2001cited by this paper
• Regressions by Leaps and Bounds

  2000cited by this paper
• Atomic Decomposition by Basis Pursuit

  1998cited by this paper
• Regression Shrinkage and Selection via the Lasso

  1996influential reference
• Sparse Approximate Solutions to Linear Systems

  1995cited by this paper
• Adaptive time-frequency decompositions with matching pursuit

  1994cited by this paper
• Matching pursuits with time-frequency dictionaries

  1993cited by this paper
• The discarding of variables in multivariate analysis.

  1967cited by this paper
• Selection of the Best Subset in Regression Analysis

  1967cited by this paper
• Applied Regression Analysis

  1966cited by this paper

• Sparsity-Driven Aggregation of Mixed Integer Programs

  2025cites this paper
• An introduction to R package 'mvs'

  2025influential citation
• Optimized data-driven method to study the self-healing and durability of ultra-high performance concrete

  2025cites this paper
• Modeling and Variable Selection of Healthcare Trip Behaviours Using Statistical Learning Techniques

  2025cites this paper
• Simulating nanoparticle concentration using stochastic models to improve indoor air quality in the industry

  2025cites this paper
• Variable Selection Methods for Multivariate, Functional, and Complex Biomedical Data in the AI Age

  2025cites this paper
• Probabilistic and nonlinear compressive sensing

  2025influential citation
• A beginner's guide to supervised analysis for mass cytometry data in cancer biology

  2024cites this paper
• Solution path algorithm for distributionally robust regression

  2024cites this paper
• Randomization Can Reduce Both Bias and Variance: A Case Study in Random Forests

  2024cites this paper
• Detecting interactions using Bayesian additive regression trees

  2024cites this paper
• Theoretical Analysis of Leave-one-out Cross Validation for Non-differentiable Penalties under High-dimensional Settings

  2024cites this paper
• A parametric approach for solving convex quadratic optimization with indicators over trees

  2024cites this paper
• Logistic Regression makes small LLMs strong and explainable "tens-of-shot" classifiers

  2024cites this paper
• A data science approach to optimize ADHD assessment with the BRIEF-2 questionnaire

  2024cites this paper
• Using Regularized Methods to Validate Q-Matrix in Cognitive Diagnostic Assessment

  2024cites this paper
• Are toe fringes important for lizard burying in highly mobile sand?

  2024cites this paper
• Performance Evaluation of Regression-Based Machine Learning Models for Modeling Reference Evapotranspiration with Temperature Data

  2024cites this paper
• The Use of Machine Learning in the Diagnosis of Kidney Allograft Rejection: Current Knowledge and Applications

  2024cites this paper
• Assessing digital transformation readiness: a comprehensive study of local clinics in Northwest Vietnam

  2024cites this paper
• Prognostic Model and Immune Response of Clear Cell Renal Cell Carcinoma Based on Co-Expression Genes Signature.

  2024cites this paper
• Evaluating Intersectional Fairness in Algorithmic Decision Making Using Intersectional Differential Algorithmic Functioning

  2024cites this paper
• Classical Statistical (In-Sample) Intuitions Don't Generalize Well: A Note on Bias-Variance Tradeoffs, Overfitting and Moving from Fixed to Random Designs

  2024cites this paper
• Dynamic Incremental Optimization for Best Subset Selection

  2024cites this paper
• A review of geospatial exposure models and approaches for health data integration

  2024cites this paper
• Sparse-ProxSkip: Accelerated Sparse-to-Sparse Training in Federated Learning

  2024influential citation
• Safe Peeling for $\ell_{0}$-Regularized Least-Squares

  2023cites this paper
• QTL analysis of important agronomic traits and metabolites in foxtail millet (Setaria italica) by RIL population and widely targeted metabolome

  2023cites this paper
• Comparison of likelihood penalization and variance decomposition approaches for clinical prediction models: A simulation study

  2023cites this paper
• Knowledge-Informed Sparse Learning for Relevant Feature Selection and Optimal Quality Prediction

  2023cites this paper
• Is there a change in the appropriateness of admission after patients were admitted? Evidence from four county hospitals in rural China

  2023cites this paper
• Exact recovery of Granger causality graphs with unconditional pairwise tests

  2023cites this paper
• Variable selection in additive models via hierarchical sparse penalty

  2023cites this paper
• Safe peeling for l0-regularized least-squares with supplementary material

  2023cites this paper
• Sparse Models for Machine Learning

  2023cites this paper
• Best Subset Selection with Efficient Primal-Dual Algorithm

  2022cites this paper
• LASSO-Based Pattern Recognition for Replenished Items With Graded Responses in Multidimensional Computerized Adaptive Testing

  2022cites this paper
• Vector textures derived from higher order derivative domains for classification of colorectal polyps

  2022cites this paper
• Compressive spectrum sensing for 5G cognitive radio networks – LASSO approach

  2022cites this paper
• Sparse index tracking using sequential Monte Carlo

  2022cites this paper
• ControlBurn: Nonlinear Feature Selection with Sparse Tree Ensembles

  2022cites this paper
• Sleep and Circadian Health of Critical COVID-19 Survivors 3 Months After Hospital Discharge

  2022cites this paper
• Machine learning based data driven inkjet printed electronics: jetting prediction for novel inks

  2022cites this paper
• Knowledge and Response to the COVID-19 Pandemic in People With Severe Mental Illness in Bangladesh and Pakistan: A Cross-Sectional Survey

  2022cites this paper
• Using Lasso and Adaptive Lasso to Identify DIF in Multidimensional 2PL Models

  2022cites this paper
• Relaxed Adaptive Lasso and Its Asymptotic Results

  2022cites this paper
• Predicting Default Probabilities for Stress Tests: A Comparison of Models

  2022cites this paper
• Defining Models to Classify between Benign and Malignant Adnexal Masses Using Routine Laboratory Parameters

  2022cites this paper
• An Alternating Method for Cardinality-Constrained Optimization: A Computational Study for the Best Subset Selection and Sparse Portfolio Problems

  2022cites this paper
• Stream water quality prediction using boosted regression tree and random forest models

  2022influential citation
• Sparsifying the least-squares approach to PCA: comparison of lasso and cardinality constraint

  2022cites this paper
• Big Data Meets the Turbulent

  2022cites this paper
• Suicide risk stratification among major depressed patients based on a machine learning approach and whole-brain functional connectivity.

  2022cites this paper
• An Approximated Collapsed Variational Bayes Approach to Variable Selection in Linear Regression

  2022influential citation
• Develop a diagnostic tool for dementia using machine learning and non-imaging features

  2022cites this paper
• Adaptive Iterative Hard Thresholding for Least Absolute Deviation Problems with Sparsity Constraints

  2022cites this paper
• Association between chronic low back pain and regular exercise, sedentary behaviour and mental health before and during COVID-19 pandemic: insights from a large-scale cross-sectional study in Germany

  2022cites this paper
• Leveraged least trimmed absolute deviations

  2021cites this paper
• Principal Component Hierarchy for Sparse Quadratic Programs

  2021cites this paper
• Sensor virtual para classificação de emissões de so2 baseado em k-NN (k-Nearest Neighbors)

  2021influential citation
• LASSO+DEA for small and big wide data

  2021cites this paper
• SELECTION OF SNP MARKERS: ANALYZING GAW17 DATA USING DIFFERENT METHODOLOGIES

  2021cites this paper
• Assessing time dependent changes in microbial composition of biological crime scene traces using microbial RNA markers.

  2021cites this paper
• Staff Level and Healthcare Access Equality in Long-Term Care Settings: Evidence from Regulation on Mandatory Nurses Overtime

  2021cites this paper
• Grafted and vanishing random subspaces

  2021influential citation
• Identification of high‐dimensional omics‐derived predictors for tumor growth dynamics using machine learning and pharmacometric modeling

  2021cites this paper
• How do macroeconomic news surprises affect round-the-clock price discovery of gold?

  2021cites this paper
• A sparse regression and neural network approach for financial factor modeling

  2021cites this paper
• Sparse Regression in Cancer Genomics: Comparing Variable Selection and Predictions in Real World Data

  2021cites this paper
• When Choosing the Best Subset Is Not the Best Choice

  2021cites this paper
• Cross-Fitted Residual Regression for High-Dimensional Heteroscedasticity Pursuit

  2021cites this paper
• Robust Variable Selection with Optimality Guarantees for High-Dimensional Logistic Regression

  2021influential citation
• THE HOMOPHONE DOMINANCE EFFECT IN REAL-LIFE DATA 1 From experiment to real-life data: social factors determine the rate of spelling errors on rule-governed verb homophones but not the size of the homophone dominance effect

  2021cites this paper
• Risk factors associated with pulmonary hypertension in obesity hypoventilation syndrome

  2021cites this paper
• A Mixed-Integer Fractional Optimization Approach to Best Subset Selection

  2021influential citation
• The impact of analogical effects and social factors on the spelling of partially homophonous verb forms in informal social media writing

  2021cites this paper
• Forward stability and model path selection

  2021cites this paper
• Development of an ensemble machine learning prognostic model to predict 60-day risk of major adverse cardiac events in adults with chest pain

  2021cites this paper
• Combination of inflammatory and vascular markers in the febrile phase of dengue is associated with more severe outcomes

  2021cites this paper
• Covariate Selection for Estimating Individual Treatment Effects in Psychotherapy Research: A Simulation Study and Empirical Example

  2021cites this paper
• Influence of clinical setting and cat characteristics on indirectly measured blood pressure and pulse rate in healthy Birman, Norwegian Forest, and Domestic Shorthair cats

  2021cites this paper
• Exterior-point Optimization for Nonconvex Learning

  2021cites this paper
• Differentially private methods for managing model uncertainty in linear regression models

  2021cites this paper
• Performance Prediction Model of Dynamic Pressure Oil-Air Separator

  2021cites this paper
• A Comparison of Feature Selection and Forecasting Machine Learning Algorithms for Predicting Glycaemia in Type 1 Diabetes Mellitus

  2021cites this paper
• Sparse Poisson regression via mixed-integer optimization

  2021cites this paper
• Integrating prediction in mean-variance portfolio optimization

  2021cites this paper
• The impact of the pandemic declaration on public awareness and behavior: Focusing on COVID-19 google searches

  2021cites this paper
• Extending greedy feature selection algorithms to multiple solutions

  2021cites this paper
• Predicting the Oil Market

  2021cites this paper
• Distributionally Robust Learning

  2021cites this paper
• Pulmonary Function and Radiologic Features in Survivors of Critical COVID-19

  2021cites this paper
• An attention algorithm for solving large scale structured $$l_{0}$$ l 0 -norm penalty estimation problems

  2021cites this paper
• Covariate Selection for Mortgage Default Analysis Using Survival Models

  2021influential citation
• Semi-automated simultaneous predictor selection for regression-SARIMA models

  2020cites this paper
• Stochastic Discrete First-Order Algorithm for Feature Subset Selection

  2020influential citation
• Modeling and Forecasting Gender-Based Violence through Machine Learning Techniques

  2020cites this paper
• Comparing methods for estimating patient‐specific treatment effects in individual patient data meta‐analysis

  2020cites this paper
• A Critical Review of LASSO and Its Derivatives for Variable Selection Under Dependence Among Covariates

  2020influential citation
• From experiment to real-life data

  2020cites this paper