Calibration after bootstrap for accurate uncertainty quantification in regression models

Obtaining accurate estimates of machine learning model uncertainties on newly predicted data is essential for understanding the accuracy of the model and whether its predictions can be trusted. A common approach to such uncertainty quantification is to estimate the variance from an ensemble of models, which are often generated by the generally applicable bootstrap method. In this work, we demonstrate that the direct bootstrap ensemble standard deviation is not an accurate estimate of uncertainty but that it can be simply calibrated to dramatically improve its accuracy. We demonstrate the effectiveness of this calibration method for both synthetic data and numerous physical datasets from the field of Materials Science and Engineering. The approach is motivated by applications in physical and biological science but is quite general and should be applicable for uncertainty quantification in a wide range of machine learning regression models.

• Publication year

  2022

• Venue

  npj Computational Materials

• Publication date

  2022-05-20

• Fields of study

  Not labeled

• Identifiers
  DOI 10.1038/s41524-022-00794-8
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Calibrated uncertainty for molecular property prediction using ensembles of message passing neural networks

  2021cited by this paper
• An introduction to statistical learning with applications in R

  2021cited by this paper
• Differentiable sampling of molecular geometries with uncertainty-based adversarial attacks

  2021cited by this paper
• Role of uncertainty estimation in accelerating materials development via active learning

  2020cited by this paper
• Opportunities and Challenges for Machine Learning in Materials Science

  2020cited by this paper
• Uncertainty Quantification Using Neural Networks for Molecular Property Prediction

  2020influential reference
• Error assessment and optimal cross-validation approaches in machine learning applied to impurity diffusion

  2019cited by this paper
• Multivariate Adaptive Regression Splines

  2019cited by this paper
• SciPy 1.0: fundamental algorithms for scientific computing in Python

  2019cited by this paper
• Molecular Similarity-Based Domain Applicability Metric Efficiently Identifies Out-of-Domain Compounds

  2019cited by this paper
• Evaluating and Calibrating Uncertainty Prediction in Regression Tasks

  2019cited by this paper
• A Quantitative Uncertainty Metric Controls Error in Neural Network-Driven Chemical Discovery

  2019influential reference
• Machine learning assisted design of high entropy alloys with desired property

  2019cited by this paper
• Machine Learning With Python

  2019cited by this paper
• Methods for comparing uncertainty quantifications for material property predictions

  2019cited by this paper
• A Unified Framework for Random Forest Prediction Error Estimation

  2019cited by this paper
• The Materials Simulation Toolkit for Machine learning (MAST-ML): An automated open source toolkit to accelerate data-driven materials research

  2019cited by this paper
• Fast and Accurate Uncertainty Estimation in Chemical Machine Learning.

  2018cited by this paper
• Accurate Uncertainties for Deep Learning Using Calibrated Regression

  2018cited by this paper
• Molecular Similarity-Based Domain Applicability Metric Efficiently Identifies Out-of-Domain Compounds.

  2018cited by this paper
• Predicting the thermodynamic stability of perovskite oxides using machine learning models

  2018cited by this paper
• Understanding the physical metallurgy of the CoCrFeMnNi high-entropy alloy: an atomistic simulation study

  2018cited by this paper
• Accelerated Discovery of Large Electrostrains in BaTiO3‐Based Piezoelectrics Using Active Learning

  2018cited by this paper
• Machine learning modeling of superconducting critical temperature

  2017cited by this paper
• High-Dimensional Materials and Process Optimization Using Data-Driven Experimental Design with Well-Calibrated Uncertainty Estimates

  2017cited by this paper
• Machine learning bandgaps of double perovskites

  2016cited by this paper
• TensorFlow: A system for large-scale machine learning

  2016cited by this paper
• A database to enable discovery and design of piezoelectric materials

  2015cited by this paper
• Charting the complete elastic properties of inorganic crystalline compounds

  2015cited by this paper
• An Introduction to Statistical Learning

  2013cited by this paper
• Confidence intervals for random forests: the jackknife and the infinitesimal jackknife

  2013cited by this paper
• Neural representations of events arise from temporal community structure

  2013cited by this paper
• Scikit-learn: Machine Learning in Python

  2011cited by this paper
• Standard errors for bagged and random forest estimators

  2009cited by this paper
• JACKKNIFE , BOOTSTRAP AND OTHER RESAMPLING METHODS IN REGRESSION ANALYSIS ' BY

  2008cited by this paper
• Prediction error estimation: a comparison of resampling methods

  2005cited by this paper
• Random Forests

  2001cited by this paper
• Probabilistic Outputs for Support vector Machines and Comparisons to Regularized Likelihood Methods

  1999cited by this paper
• Efficient Global Optimization of Expensive Black-Box Functions

  1998cited by this paper
• Improvements on Cross-Validation: The 632+ Bootstrap Method

  1997cited by this paper
• A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection

  1995cited by this paper
• Jackknife‐After‐Bootstrap Standard Errors and Influence Functions

  1992cited by this paper
• Discussion: Jackknife, Bootstrap and Other Resampling Methods in Regression Analysis

  1986cited by this paper
• A Leisurely Look at the Bootstrap, the Jackknife, and

  1983cited by this paper

• A multi-fidelity active learning framework for accelerated alloy design

  2026cites this paper
• Predictors of past 30-day vaping abstinence among young e-cigarette users: Machine learning analysis of a longitudinal cohort.

  2026cites this paper
• Uncertainty estimation affects predictor selection and its calibration improves materials optimization

  2026cites this paper
• A Critical Examination of Active Learning Workflows in Materials Science

  2026cites this paper
• Probabilistic modeling of microwave thermal damage at rough rock-cement interfaces using a Bayesian-ANN framework

  2026cites this paper
• A transfer-learning framework to alleviate data scarcity in cross-slope wind pressure modeling

  2026cites this paper
• Robust learning from literature data: Model generalizability and uncertainty for predicting conjugated polymer solution conformation

  2026cites this paper
• A spatiotemporally coordinated strategy for active and deterministic edge control in bonnet polishing

  2026cites this paper
• Explainable machine learning for predicting tensile properties of aluminum alloys in the laser powder bed fusion process

  2025cites this paper
• Materials-discovery workflow guided by symbolic regression for identifying acid-stable oxides for electrocatalysis

  2025cites this paper
• Machine Learning-Guided Discovery of Temperature-Induced Solid-Solid Phase Transitions in Inorganic Materials

  2025cites this paper
• Towards a Comprehensive Taxonomy of Pointwise Reliability Assessment Techniques for Machine Learning Models

  2025cites this paper
• A framework for the dynamic identification of drought propensity: The predictive role of energy changes in drought events

  2025cites this paper
• Prediction of Unconfined Compressive Strength in Cement-Treated Soils: A Machine Learning Approach

  2025cites this paper
• Active learning-enhanced yttrium oxyhydride thin films for photoinduced insulator-to-metal transitions

  2025cites this paper
• A Critical Review of Machine Learning Analyses of Reactor Pressure Vessel Embrittlement and Hardening

  2025cites this paper
• Machine learning for modeling intergranular stress corrosion cracking of stainless steels in light water reactors with uncertainty quantification and explainability

  2025cites this paper
• Hybrid data-driven framework for interpretable prediction of nitrate and sulfate risks in coastal aquifers.

  2025cites this paper
• Building Trustworthy AI for Materials Discovery: From Autonomous Laboratories to Z-scores

  2025cites this paper
• Applications of Machine Learning in High-Entropy Alloys: Phase Prediction, Performance Optimization, and Compositional Space Exploration

  2025cites this paper
• When Active Learning Fails, Uncalibrated Out of Distribution Uncertainty Quantification Might Be the Problem

  2025cites this paper
• Defect Diffusion Graph Neural Networks for Materials Discovery in High-Temperature Energy Applications

  2025cites this paper
• A machine learning approach for estimating forage maize yield and quality in NW Spain

  2025cites this paper
• Improving explainable AI in attributing hydrological responses to climate variabilities in snow-dominated watersheds

  2025cites this paper
• Predicting fall parameters from infant skull fractures using machine learning

  2025cites this paper
• Learning atomic forces from uncertainty-calibrated adversarial attacks

  2025cites this paper
• State Estimation of Lithium-Ion Batteries via Physics-Machine Learning Combined Methods: A Methodological Review and Future Perspectives

  2025cites this paper
• The synchronized dynamic release behavior of microplastics during farmland soil erosion process.

  2024cites this paper
• Negative impact of heavy-tailed uncertainty and error distributions on the reliability of calibration statistics for machine learning regression tasks

  2024cites this paper
• Validation of ML-UQ calibration statistics using simulated reference values: a sensitivity analysis

  2024cites this paper
• Coherent collections of rules describing exceptional materials identified with a multi-objective optimization of subgroups

  2024cites this paper
• Ensemble learning for predicting average thermal extraction load of a hydrothermal geothermal field: A case study in Guanzhong Basin, China

  2024cites this paper
• Accelerating Ensemble Error Bar Prediction with Single Models Fits

  2024cites this paper
• Analysis of machine learning prediction reliability based on sampling distance evaluation with feature decorrelation

  2024cites this paper
• Chemoinformatic regression methods and their applicability domain

  2024cites this paper
• A general approach for determining applicability domain of machine learning models

  2024cites this paper
• Machine learning materials properties with accurate predictions, uncertainty estimates, domain guidance, and persistent online accessibility

  2024cites this paper
• Power Control for NN-Based Wireless Distributed Inference With Improved Model Calibration

  2024cites this paper
• Uncertainty Quantification in Predicting Physical Property of Porous Medium With Bayesian Evidential Learning

  2024influential citation
• Recent progress on machine learning with limited materials data: Using tools from data science and domain knowledge

  2024cites this paper
• M\"obius inversion and the iterated bootstrap

  2024cites this paper
• Machine learning in materials research: Developments over the last decade and challenges for the future

  2024cites this paper
• Predictive Modeling of High-Entropy Alloys and Amorphous Metallic Alloys Using Machine Learning

  2024cites this paper
• Materials-Discovery Workflows Guided by Symbolic Regression: Identifying Acid-Stable Oxides for Electrocatalysis

  2024cites this paper
• MatFold: systematic insights into materials discovery models’ performance through standardized cross-validation protocols

  2024cites this paper
• Implementing Trust in Non-Small Cell Lung Cancer Diagnosis with a Conformalized Uncertainty-Aware AI Framework in Whole-Slide Images

  2024cites this paper
• Multivariate prediction intervals for bagged models

  2023cites this paper
• Exploring the uncertainty of machine learning models and geostatistical mapping of rare earth element potential in Indiana coals, USA

  2023cites this paper
• Machine Learning Design of Perovskite Catalytic Properties

  2023cites this paper
• Comment on ‘Physics-based representations for machine learning properties of chemical reactions’

  2023cites this paper
• Current Methods for Drug Property Prediction in the Real World

  2023cites this paper
• Estimation of cardiorespiratory fitness using heart rate and step count data

  2023cites this paper
• Calibration in machine learning uncertainty quantification: Beyond consistency to target adaptivity

  2023cites this paper
• Predictions and Uncertainty Estimates of Reactor Pressure Vessel Steel Embrittlement Using Machine Learning

  2023cites this paper
• Weight Uncertainty in Transformer Network for the Traveling Salesman Problem

  2023cites this paper
• Deep Learning Framework for Lithium-ion Battery State of Charge Estimation: Recent Advances and Future Perspectives

  2023cites this paper
• Properties of the ENCE and other MAD-based calibration metrics

  2023cites this paper
• Materials Informatics: An Algorithmic Design Rule

  2023cites this paper
• Validation of uncertainty quantification metrics: a primer based on the consistency and adaptivity concepts

  2023cites this paper
• Confidence curves for UQ validation: probabilistic reference vs. oracle

  2022cites this paper
• Fast Uncertainty Estimates in Deep Learning Interatomic Potentials

  2022cites this paper
• Predicting binding motifs of complex adsorbates using machine learning with a physics-inspired graph representation

  2022cites this paper