Computing AIC for black-box models using generalized degrees of freedom: A comparison with cross-validation

ABSTRACT Generalized degrees of freedom (GDF), as defined by Ye (1998 JASA 93:120–131), represent the sensitivity of model fits to perturbations of the data. Such GDF can be computed for any statistical model, making it possible, in principle, to derive the effective number of parameters in machine-learning approaches and thus compute information-theoretical measures of fit. We compare GDF with cross-validation and find that the latter provides a less computer-intensive and more robust alternative. For Bernoulli-distributed data, GDF estimates were unstable and inconsistently sensitive to the number of data points perturbed simultaneously. Cross-validation, in contrast, performs well also for binary data, and for very different machine-learning approaches.

• Publication year

  2016

• Venue

  Communications in statistics. Simulation and computation

• Publication date

  2016-03-09

• Fields of study

  Mathematics, Computer Science

• Identifiers
  DOI 10.1080/03610918.2017.1315728arXiv 1603.02743
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Behavioral Ecology and Sociobiology

  2019cited by this paper
• R: A language and environment for statistical computing.

  2014influential reference
• Generalized degrees of freedom and adaptive model selection in linear mixed-effects models

  2012cited by this paper
• Assessment and Propagation of Model Uncertainty

  2011cited by this paper
• Where’s the sperm whale? A species distribution example analysis

  2011cited by this paper
• Species Distribution Models: Ecological Explanation and Prediction Across Space and Time

  2010cited by this paper
• Using information theory as a substitute for stepwise regression in ecology and behavior

  2010cited by this paper
• Issues in information theory-based statistical inference—a commentary from a frequentist’s perspective

  2010cited by this paper
• Modern Applied Statistics with S

  2010cited by this paper
• Evolution of climate niches in European mammals?

  2010cited by this paper
• Model selection and information theory in geographical ecology

  2008cited by this paper
• Machine Learning Methods Without Tears: A Primer for Ecologists

  2008cited by this paper
• Generalized Additive Models: An Introduction With R

  2007influential reference
• Classification and Regression by randomForest

  2007influential reference
• Likelihood Cross-Validation Versus Least Squares Cross-Validation for Choosing the Smoothing Parameter in Kernel Home-Range Analysis

  2006cited by this paper
• Optimal Model Assessment, Selection, and Combination

  2006cited by this paper
• Novel methods improve prediction of species' distributions from occurrence data

  2006cited by this paper
• The elements of statistical learning: data mining, inference and prediction

  2005influential reference
• The Elements of Statistical Learning: Data Mining, Inference, and Prediction

  2004cited by this paper
• The Problem of Overfitting

  2004cited by this paper
• Adaptive Model Selection and Assessment for Exponential Family Distributions

  2004cited by this paper
• Model selection and multimodel inference : a practical information-theoretic approach

  2003cited by this paper
• Model Selection and Multimodel Inference: A Practical Information-Theoretic Approach (2nd ed.)

  2003cited by this paper
• The Generalization Paradox of Ensembles

  2003influential reference
• On the Undecidability among Kinetic Models: From Model Selection to Model Averaging

  2003cited by this paper
• Bayesian measures of model complexity and fit

  2002cited by this paper
• Applications of machine learning to ecological modelling

  2001cited by this paper
• On Measuring and Correcting the Effects of Data Mining and Model Selection

  1998influential reference
• On Information and Sufficiency

  1997cited by this paper
• Approximate Bayes factors and accounting for model uncertainty in generalised linear models

  1996cited by this paper
• A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection

  1995cited by this paper
• Bayesian Graphical Models for Discrete Data

  1995cited by this paper
• Evaluation of the Kullback-Leibler Discrepancy for Model Selection in Open Population Capture-Recapture Models

  1994cited by this paper
• Regression and time series model selection in small samples

  1989cited by this paper
• Further analysts of the data by akaike' s information criterion and the finite corrections

  1978cited by this paper
• An Asymptotic Equivalence of Choice of Model by Cross‐Validation and Akaike's Criterion

  1977cited by this paper
• Information Theory and an Extension of the Maximum Likelihood Principle

  1973influential reference

• Measuring Neural Network Complexity via Effective Degrees of Freedom

  2026influential citation
• Harnessing ensemble Machine learning models for improved salinity prediction in large river basin scales

  2025cites this paper
• The probability of improved prediction as a new concept for model selection in the presence of outliers

  2025cites this paper
• Distilling knowledge from graph neural networks trained on cell graphs to non-neural student models

  2025influential citation
• Data-driven model for marine engine fault diagnosis using in-cylinder pressure signals

  2024cites this paper
• Polishing the black box: flexible model-based partitioning surrogate models for interpretable machine learning model

  2024cites this paper
• Flexible species distribution modelling methods perform well on spatially separated testing data

  2023cites this paper
• Modeling the size of co-seismic landslides via data-driven models: the Kaik¯oura’s example

  2022cites this paper
• Cognitive Performance and Learning Parameters Predict Response to Working Memory Training in Parkinson’s Disease

  2022cites this paper
• Daily 1 km terrain resolving maps of surface fine particulate matter for the western United States 2003–2021

  2022cites this paper
• Resource selection functions based on hierarchical generalized additive models provide new insights into individual animal variation and species distributions

  2021cites this paper
• Influence of Irrigation Drivers Using Boosted Regression Trees: Kansas High Plains

  2021cites this paper
• Predict long-range enhancer regulation based on protein–protein interactions between transcription factors

  2021influential citation
• Deep Learning With Anaphora Resolution for the Detection of Tweeters With Depression: Algorithm Development and Validation Study

  2020cites this paper
• Interpretable Machine Learning - A Brief History, State-of-the-Art and Challenges

  2020cites this paper
• Estimation of the forest stand mean height and aboveground biomass in Northeast China using SAR Sentinel-1B, multispectral Sentinel-2A, and DEM imagery

  2019cites this paper
• Web Scraping Crawling-based Automatic Data Augmentation for Deep Neural Networks-based Vehicle Classifications

  2019cites this paper
• Limitations of Species Distribution Models Based on Available Climate Change Data: A Case Study in the Azorean Forest

  2019cites this paper
• Frequentist Model Averaging

  2018cites this paper
• Optimizing ensembles of small models for predicting the distribution of species with few occurrences

  2018cites this paper
• Model averaging in ecology: a review of Bayesian, information-theoretic, and tactical approaches for predictive inference

  2018cites this paper