Improving the Estimation of Prediction Increment Measures in Logistic and Survival Analysis

Simple Summary Confidence interval estimation of discrimination improvement measures, including the area under the receiver operating characteristic curve, the net reclassification index, and the integrated discrimination improvement, is an area of ongoing research. The most common confidence interval estimation methods employ normal theory. Literature suggests that degeneration of the normal assumption under the null hypothesis exists, and normal theory confidence intervals estimated may be invalid. Bootstrapped confidence intervals do not rely on normal theory assumptions. We examine the performance of discrimination improvement measures in both the logistic and survival regression context through simulation. Normal theory intervals are only appropriate with a strong effect size of the added parameter, and the percentile bootstrap interval exhibits reasonable coverage while maintaining the shortest width in nearly all simulated scenarios, making this interval the most reliable choice. The intent is that these recommendations improve the accuracy in the estimation and the overall assessment of discrimination improvement.

• Publication year

  2025

• Venue

  Cancers

• Publication date

  2025-04-01

• Fields of study

  Mathematics, Medicine

• Identifiers
  DOI 10.3390/cancers17081259PMID 40282435PMCID 12025450
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar, PubMed

• No claims are published for this paper.

• No concepts are published for this paper.

• Assessment of common risk factors and validation of the Gail model for breast cancer: A hospital-based study from Western India

  2020cited by this paper
• Prospective validation of the NCI Breast Cancer Risk Assessment Tool (Gail Model) on 40,000 Australian women

  2018cited by this paper
• Net reclassification index at event rate: properties and relationships

  2017influential reference
• An Adaptive Resampling Test for Detecting the Presence of Significant Predictors

  2015cited by this paper
• A note on the evaluation of novel biomarkers: do not rely on integrated discrimination improvement and net reclassification index

  2014cited by this paper
• Net reclassification indices for evaluating risk prediction instruments: a critical review.

  2014cited by this paper
• Calibration of models is not sufficient to justify NRI

  2014cited by this paper
• Net reclassification improvement and integrated discrimination improvement require calibrated models: relevance from a marker and model perspective

  2014cited by this paper
• Testing for improvement in prediction model performance

  2013cited by this paper
• Goodness-of-Fit Tests and Model Validity

  2012cited by this paper
• Misuse of DeLong test to compare AUCs for nested models

  2012cited by this paper
• Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers

  2011influential reference
• Evaluating the incremental value of new biomarkers with integrated discrimination improvement.

  2011cited by this paper
• Practice of Epidemiology Problems With Risk Reclassification Methods for Evaluating Prediction Models

  2011cited by this paper
• Re: "problems with risk reclassification methods for evaluating prediction models".

  2011cited by this paper
• Several methods to assess improvement in risk prediction models: Extension to survival analysis

  2011influential reference
• Clinical Prediction Models—a Practical Approach to Development, Validation and Updating

  2009cited by this paper
• Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond

  2008influential reference
• Use and Misuse of the Receiver Operating Characteristic Curve in Risk Prediction

  2007cited by this paper
• An Introduction to the Bootstrap

  2007influential reference
• Prognostic/Clinical Prediction Models: Multivariable Prognostic Models: Issues in Developing Models, Evaluating Assumptions and Adequacy, and Measuring and Reducing Errors

  2005cited by this paper
• When is a new prediction marker useful? A consideration of lipoprotein-associated phospholipase A2 and C-reactive protein for stroke risk.

  2005cited by this paper
• Overall C as a measure of discrimination in survival analysis: model specific population value and confidence interval estimation

  2004influential reference
• Regression Modeling Strategies

  2003cited by this paper
• Validation of the Gail et al. model of breast cancer risk prediction and implications for chemoprevention.

  2001cited by this paper
• Regression Modeling Strategies: With Applications to Linear Models, Logistic Regression, and Survival Analysis

  2001influential reference
• Validation of the Gail et al. model for predicting individual breast cancer risk.

  1994cited by this paper
• Projecting individualized probabilities of developing breast cancer for white females who are being examined annually.

  1989cited by this paper
• Theoretical Comparison of Bootstrap Confidence Intervals

  1988cited by this paper
• Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach.

  1988influential reference
• Nonparametric standard errors and confidence intervals

  1981cited by this paper

• No citing papers are available for this paper.