A survey of cross-validation procedures for model selection

Used to estimate the risk of an estimator or to perform model selection, cross-validation is a widespread strategy because of its simplicity and its apparent universality. Many results exist on the model selection performances of cross-validation procedures. This survey intends to relate these results to the most recent advances of model selection theory, with a particular emphasis on distinguishing empirical statements from rigorous theoretical results. As a conclusion, guidelines are provided for choosing the best cross-validation procedure according to the particular features of the problem in hand.

• Publication year

  2009

• Venue

  Statistics Surveys

• Publication date

  2009-07-27

• Fields of study

  Mathematics, Computer Science

• Identifiers
  DOI 10.1214/09-SS054arXiv 0907.4728
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Statistical Learning Theory

  2021cited by this paper
• Classification and regression trees

  2012cited by this paper
• Statistical Applications in Genetics and Molecular Biology Asymptotic Optimality of Likelihood-Based Cross-Validation

  2011cited by this paper
• Minimum Description Length Principle.

  2010cited by this paper
• Adaptive and Learning Systems for Signal Processing, Communication, and Control

  2010cited by this paper
• Model selection by resampling penalization

  2009influential reference
• Histogram selection in non gaussian regression

  2009cited by this paper
• Linear Model Selection by Cross-validation

  2009cited by this paper
• Density estimation via cross-validation: Model selection point of view

  2009influential reference
• Data-driven Calibration of Penalties for Least-Squares Regression

  2008cited by this paper
• Nonparametric density estimation by exact leave-p-out cross-validation

  2008influential reference
• Model selection in density estimation via cross-validation

  2008influential reference
• Catching Up Faster by Switching Sooner: A Prequential Solution to the AIC-BIC Dilemma

  2008cited by this paper
• Estimating the Error Rate of a Prediction Rule: Improvement on Cross-Validation

  2008cited by this paper
• V-fold cross-validation improved: V-fold penalization

  2008influential reference
• Least angle and ℓ1 penalized regression: A review

  2008cited by this paper
• Suboptimality of penalties proportional to the dimension for model selection in heteroscedastic regression

  2008influential reference
• Estimation of the conditional risk in classification: The swapping method

  2008cited by this paper
• Model selection via cross-validation in density estimation, regression, and change-points detection

  2008influential reference
• Concentration inequalities and model selection

  2007cited by this paper
• Suboptimality of Penalized Empirical Risk Minimization in Classification

  2007cited by this paper
• CONSISTENCY OF CROSS VALIDATION FOR COMPARING REGRESSION PROCEDURES

  2007influential reference
• Minimal Penalties for Gaussian Model Selection

  2007influential reference
• Oracle inequalities for multi-fold cross validation

  2006cited by this paper
• Local Rademacher complexities and oracle inequalities in risk minimization

  2006cited by this paper
• Discussion: Local Rademacher complexities and oracle inequalities in risk minimization

  2006influential reference
• COMPARING LEARNING METHODS FOR CLASSIFICATION

  2006cited by this paper
• The Cross-Validated Adaptive Epsilon-Net Estimator

  2006cited by this paper
• Adaptive and Learning Systems for Signal Processing, Communications, and Control

  2006cited by this paper
• Optimal Oracle Inequality for Aggregation of Classifiers Under Low Noise Condition

  2006cited by this paper
• Estimation of Dependences Based on Empirical Data

  2006cited by this paper
• Local Rademacher complexities

  2005influential reference
• Analysis of Variance of Cross-Validation Estimators of the Generalization Error

  2005cited by this paper
• Cross-validation in nonparametric regression with outliers

  2005cited by this paper
• Exact oracle inequality for a sharp adaptive kernel density estimator

  2005cited by this paper
• Cross validation model selection criteria for linear regression based on the Kullback–Leibler discrepancy

  2005cited by this paper
• Prediction error estimation: a comparison of resampling methods

  2005influential reference
• Can the Strengths of AIC and BIC Be Shared

  2005cited by this paper
• Theory of classification : a survey of some recent advances

  2005cited by this paper
• Model selection by bootstrap penalization for classification

  2004cited by this paper
• The Elements of Statistical Learning

  2003influential reference
• Rademacher and Gaussian Complexities: Risk Bounds and Structural Results

  2003cited by this paper
• Unified Cross-Validation Methodology For Selection Among Estimators and a General Cross-Validated Adaptive Epsilon-Net Estimator: Finite Sample Oracle Inequalities and Examples

  2003cited by this paper
• Model selection for regression on a random design

  2002cited by this paper
• Stability and Generalization

  2002cited by this paper
• A Distribution-Free Theory of Nonparametric Regression (Springer Series in Statistics)

  2002cited by this paper
• A Distribution-Free Theory of Nonparametric Regression

  2002cited by this paper
• Adaptive Regression by Mixing

  2001cited by this paper
• Subsampling

  2001cited by this paper
• Nonparametric Regression with Correlated Errors

  2001cited by this paper
• Rademacher penalties and structural risk minimization

  2001cited by this paper
• Gaussian model selection

  2001influential reference
• Some Comments on Cp

  2000influential reference
• Topics in Non-Parametric Statistics

  2000cited by this paper
• Model Selection and Error Estimation

  2000cited by this paper
• On the use of cross-validation to assess performance in multivariate prediction

  2000cited by this paper
• Bayesian model averaging: a tutorial (with comments by M. Clyde, David Draper and E. I. George, and a rejoinder by the authors

  1999cited by this paper
• Risk bounds for model selection via penalization

  1999cited by this paper
• Combined 5 x 2 cv F Test for Comparing Supervised Classification Learning Algorithms

  1999cited by this paper
• Arcing Classifiers

  1998cited by this paper
• Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms

  1998cited by this paper
• Arcing classifier (with discussion and a rejoinder by the author)

  1998cited by this paper
• Asymptotic comparison of (partial) cross-validation, GCV and randomized GCV in nonparametric regression

  1998cited by this paper
• Robust Linear Model Selection by Cross-Validation

  1997cited by this paper
• AN ASYMPTOTIC THEORY FOR LINEAR MODEL SELECTION

  1997cited by this paper
• MODEL SELECTION FOR NONPARAMETRIC REGRESSION

  1997influential reference
• Algorithmic Stability and Sanity-Check Bounds for Leave-One-Out Cross-Validation

  1997cited by this paper
• Improvements on Cross-Validation: The 632+ Bootstrap Method

  1997cited by this paper
• Regression Shrinkage and Selection via the Lasso

  1996cited by this paper
• Bootstrap Model Selection

  1996cited by this paper
• Heuristics of instability and stabilization in model selection

  1996influential reference
• A Probabilistic Theory of Pattern Recognition

  1996influential reference
• Bayesian Model Selection in Social Research

  1995cited by this paper
• On bandwidth choice in nonparametric regression with both short- and long-range dependent errors

  1995cited by this paper
• Adaptive and learning systems for signal processing, communications and control

  1995cited by this paper
• An Experimental and Theoretical Comparison of Model Selection Methods

  1995cited by this paper
• More comments on C p

  1995cited by this paper
• Pattern Recognition and Neural Networks

  1995cited by this paper
• A cross-validatory method for dependent data

  1994cited by this paper
• Automated Kernel Smoothing of Dependent Data by Using Time Series Cross‐Validation

  1994cited by this paper
• Bandwidth selection in robust smoothing

  1993cited by this paper
• Model Selection Via Multifold Cross Validation

  1993influential reference
• DATA‐DEPENDENT ESTIMATION OF PREDICTION FUNCTIONS

  1992cited by this paper
• Smoothed cross-validation

  1992influential reference
• Submodel selection and evaluation in regression. The X-random case

  1992influential reference
• On the bias and variability of bootstrap and cross-validation estimates of error rate in discrimination problems

  1992cited by this paper
• Comparison of Two Bandwidth Selectors with Dependent Errors

  1991cited by this paper
• Statistical Design and Analysis of Experiments

  1990cited by this paper
• Data-Driven Bandwidth Choice for Density Estimation Based on Dependent Data

  1990cited by this paper
• A comparative study of ordinary cross-validation, v-fold cross-validation and the repeated learning-testing methods

  1989influential reference
• Regression and time series model selection in small samples

  1989cited by this paper
• A local cross-validation algorithm

  1989cited by this paper
• How Far are Automatically Chosen Regression Smoothing Parameters from their Optimum

  1988cited by this paper
• Asymptotic Optimality for $C_p, C_L$, Cross-Validation and Generalized Cross-Validation: Discrete Index Set

  1987cited by this paper
• On Kullback-Leibler loss and density estimation

  1987cited by this paper
• Kernel Regression Estimation Using Repeated Measurements Data

  1986cited by this paper
• How Biased is the Apparent Error Rate of a Prediction Rule

  1986cited by this paper
• Smoothing noisy data with spline functions

  1985cited by this paper
• From Stein's Unbiased Risk Estimates to the Method of Generalized Cross Validation

  1985cited by this paper
• Asymptotic Properties of Criteria for Selection of Variables in Multiple Regression

  1984cited by this paper

• Health indicator modeling leveraging time-independent and time-dependent subtasks with adaptive standardization and physics-based Bayesian optimization for aeronautical structures

  2026cites this paper
• Prolonged growth and extended subadult development in the Tyrannosaurus rex species complex revealed by expanded histological sampling and statistical modeling

  2026cites this paper
• AI-Driven Design of Sustainable Flame-Retardant Biodegradable Polymer Composites

  2026cites this paper
• Estimating cumulative input effects in annual crop production: A LASSO-based panel data approach from India

  2026cites this paper
• Assessing nitrogen levels and crop yield prediction in cotton using unmanned aerial vehicle-based multispectral remote sensing

  2026cites this paper
• The impact of K selection in K‑fold cross-validation on bias and variance in supervised learning models

  2026cites this paper
• Predicting pedestrian-level melanopic illuminance from multi-source urban data with explainable machine learning

  2026cites this paper
• Semantic Neighborhood Density and Eye Gaze Time in Human Programmer Attention

  2026cites this paper
• Trustworthy prediction of enzyme commission numbers using a hierarchical interpretable transformer

  2026cites this paper
• Machine Learning-Based Ultrasonic Sensing of Glycerol in Direct Fuel Cells

  2026cites this paper
• Model selection by cross-validation in an expectile linear regression

  2026cites this paper
• Predictive Maintenance of Mining Centrifuges Using Machine Learning and Deep Learning Models

  2026cites this paper
• Benchmarking Few-shot Transferability of Pre-trained Models with Improved Evaluation Protocols

  2026cites this paper
• Leave-One-Out Neighborhood Smoothing for Graphons: Berry-Esseen Bounds, Confidence Intervals, and Honest Tuning

  2026cites this paper
• A new compendium of models for assessing the economic impacts of climate change on agricultural systems

  2026cites this paper
• Non-parametric ground motion duration and energy-based intensity measure models: A novel stacking ensemble model using regional seismic data

  2026cites this paper
• Waveform-aware deep learning for quantitative agglutination on a low-cost lab-in-tubing microfluidic platform

  2026cites this paper
• LOO-PIT predictive model checking

  2026cites this paper
• VCDF: A Validated Consensus-Driven Framework for Time Series Causal Discovery

  2026cites this paper
• A comprehensive framework for computationally efficient system-level design optimization of machine tools

  2026cites this paper
• Smartphone-based colorimetric sensing with reference calibration and ensemble machine learning for enhanced detection of nitrite and ammonium ions.

  2026cites this paper
• TVISARS-CoV-2: a multicriteria index and urban dynamics in Minas Gerais, Brazil

  2026cites this paper
• PEtab Select: specification standard and supporting software for automated model selection

  2026cites this paper
• Nutrient recommendation model for plantain (Musa AAB) cultivation based on the analysis of soil variables in the Department of Risaralda, Colombia

  2026cites this paper
• Post-Fire Forest Pulse Recovery: Superiority of Generalized Additive Models (GAM) in Long-Term Landsat Time-Series Analysis

  2026cites this paper
• Development and validation of a neural network survival prediction model for ischemic heart disease

  2026cites this paper
• A QSAR-machine learning hybrid model for predicting the ecotoxicity of soil organic compounds and deriving thresholds

  2026cites this paper
• KNLR: A heterogeneous ensemble learner for predicting Foie gras weight grade in mule ducks (Anas platyrhynchos × Cairina moschata).

  2026cites this paper
• Optimizing food protein stability through flavonoid complexation: An efficient computational-experimental approach

  2026cites this paper
• Elucidating long-term suspended sediment dynamics in Lake Tana of Ethiopia through integrating remote sensing with multi-depth field measurements

  2026cites this paper
• On convergence of regularized covariance estimator based on modified Cholesky decomposition

  2026cites this paper
• Demographic identification of Greater Caribbean manatees via acoustic feature learning

  2026cites this paper
• Informed Machine Learning: Excess risk and generalization

  2025cites this paper
• Tactile and interpersonal sound symbolic associations in patients with multiple somatic symptoms

  2025cites this paper
• Bias as a Virtue: Rethinking Generalization under Distribution Shifts

  2025cites this paper
• Robust Cross-Validation of Predictive Models Used in Credit Default Risk

  2025cites this paper
• Stability Regularized Cross-Validation

  2025cites this paper
• Quantum information geometry meets DMRG: Uhlmann gauge improvements in computational methods

  2025cites this paper
• Automatic classification of walnut (Juglans Regia L.) species using deep learning methods

  2025cites this paper
• Prediction of concrete wear resistance under wind erosion based on the LSTM deep learning model

  2025cites this paper
• Machine Learning-Enhanced MALDI-TOF MS for Real-Time Detection of Antibiotic-Resistant E. coli in Food Processing

  2025cites this paper
• Predicting nanoscale stress-strain curves: A Gaussian processes within a Bayesian framework

  2025cites this paper
• Generalizing fuzzy k-nearest neighbor classifier using an OWA operator with a RIM quantifier

  2025cites this paper
• Machine learning-based algorithm selection for irregular three-dimensional packing in additive manufacturing

  2025cites this paper
• Joint leave-group-out cross-validation in Bayesian spatial models

  2025cites this paper
• Analyzing lump coal gas release as a source of coal mine methane: Using GC and machine learning

  2025cites this paper
• A hierarchical HGBT-based machine learning framework for predicting the power conversion efficiency of perovskite solar cells

  2025cites this paper
• A deep convolutional neural network-based keratitis detection and classification approach

  2025cites this paper
• Short-Term Solar Radiation Forecasting with Discrete Wavelet Transform Based Boosted Machine Learning Methods

  2025cites this paper
• Cherry-picking: automated model term selection for precise overlay and CD process control

  2025cites this paper
• Allometric equations to predict aboveground biomass and carbon stock of Faidherbia albida in parkland agroforestry

  2025cites this paper
• How many trees are there in the North American boreal forest?

  2025cites this paper
• Probing the dimuon channel of a Z' boson at the HL-LHC using multivariate analysis

  2025cites this paper
• Study of the influence of a biased database on the prediction of standard algorithms for selecting the best candidate for an interview

  2025cites this paper
• Predicting the descent into extremism and terrorism

  2025cites this paper
• Fully automated image quality assessment based on deep learning for carotid computed tomography angiography: A multicenter study.

  2025cites this paper
• Allometric Models to Estimate the Merchantable Wood Volume and Biomass of the Most Abundant Miombo Species in the Miombo Woodlands in Mozambique

  2025cites this paper
• Brain Cysts and Cystic Tumors Classification Based on New Deep Learning Hybrid Structure

  2025cites this paper
• Decoding the Narcissistic Brain

  2025cites this paper
• Comparative Evaluation of Automatic Detection and Classification of Daily Living Activities Using Batch Learning and Stream Learning Algorithms

  2025cites this paper
• Modeling complex systems in animal science: A pedagogical framework for new researchers

  2025cites this paper
• Enhancing the prediction accuracy of real-world seismic data using various Decision tree-based models

  2025cites this paper
• Weak Signals and Heavy Tails: Machine-learning meets Extreme Value Theory

  2025cites this paper
• Assessing twin tunnel-induced ground settlements in alluvial deposits by InSAR data

  2025cites this paper
• Combining national forest inventories reveals distinct role of climate on tree recruitment in European forests

  2025cites this paper
• Risk-based completion cost prediction approach in construction projects utilizing machine learning

  2025cites this paper
• Machine learning algorithm facilitates fast derivation of light-scattering properties of sea salt aerosol

  2025cites this paper
• Preprocessing-Free Convolutional Neural Network Model for Arrhythmia Classification Using ECG Images

  2025cites this paper
• DeepATsers: a deep learning framework for one-pot SERS biosensor to detect SARS-CoV-2 virus

  2025cites this paper
• Deep Learning-Based Framework for Efficient Design of Multicomponent High Hardness High Entropy Alloys.

  2025cites this paper
• A robust sparse identification of nonlinear dynamics approach by combining neural networks and an integral form

  2025cites this paper
• Unveiling CNS cell morphology with deep learning: A gateway to anti-inflammatory compound screening

  2025cites this paper
• Shear strength parameters prediction of rock materials using hybrid machine learning model

  2025cites this paper
• Hybrid catboost models optimized with metaheuristics for predicting shear strength in rock joints

  2025cites this paper
• The Search for the Most Informative Areas for the Binary Classification of Schizophrenia Using Resting fMRI Data Based in a Method for Extracting Functionally Homogeneous Areas

  2025cites this paper
• Choosing blocks for spatial cross-validation: lessons from a marine remote sensing case study

  2025cites this paper
• Variance-based sensitivity analysis of climate variability impact on crop yield using machine learning: A case study in Jordan

  2025cites this paper
• Reducing cross-validation variance through seed blocking in hyperparameter tuning

  2025cites this paper
• Sparklen: A Statistical Learning Toolkit for High-Dimensional Hawkes Processes in Python

  2025cites this paper
• Transfer-Modal Extraction of Surface EMG Features for Upper Limb Motor Classification

  2025cites this paper
• Fish Catch Prediction by Combining Fishing, Weather and Tidal Data

  2025cites this paper
• Attribute non-attendance in the choice experiment with machine Learning: WTP for organic apples in Germany

  2025cites this paper
• Diameter and height modeling for accurate prediction of tree size in a Douglas-fir rainforest

  2025cites this paper
• Statistical evaluation of a diversified surface solar irradiation data repository and forecasting using a recurrent neural network-hybrid model: A case study in Bhutan

  2025cites this paper
• Statistical inference in the presence of imputed survey data through regression trees and random forests

  2025cites this paper
• Machine learning methods for basal area prediction of Fagus orientalis Lipsky stands based on national forest inventory

  2025cites this paper
• EarlyStopping: Implicit Regularization for Iterative Learning Procedures in Python

  2025cites this paper
• Tensor Low-Rank Approximation via Plug-and-Play Priors for Anomaly Detection in Remote Sensing Images

  2025cites this paper
• Restarted multiple kernel algorithms with self-guiding for large-scale multi-view clustering

  2025cites this paper
• Harmonia: A Multi-Agent Reinforcement Learning Approach to Data Placement and Migration in Hybrid Storage Systems

  2025cites this paper
• Active learning of optical path classification

  2025cites this paper
• Early detection of crack vulnerability in foldable displays through critical angle curvature analysis

  2025cites this paper
• Rate of double reduction and genetic variability in yield, quality, and senescence related traits in tetraploid potato (Solanum tuberosum L.)

  2025cites this paper
• A Majority Voting Mechanism-Based Ensemble Learning Approach for Financial Distress Prediction in Indian Automobile Industry

  2025cites this paper
• Tree-based, boosting, and stacked models for accurate prediction of total organic carbon from conventional well logs

  2025cites this paper
• Insights into the metal cutting contact zone through automation and multivariate regression modelling under the framework of gear skiving

  2025cites this paper
• An Intelligent Retaining Wall Selection System Based on Stacking Integrated Model

  2025cites this paper
• Unfolding key factors of resilience in ICT cognitive-motivational engagement: Global evidence from machine learning techniques

  2025cites this paper
• Representation Learning for Tabular Data: A Comprehensive Survey

  2025cites this paper
• Stacking Weights and Model Space Selection in Frequentist Model Averaging for Benchmark Dose Estimation

  2025cites this paper