Designing a quasi-experiment to study the clinical impact of adaptive risk prediction models

Clinical risk prediction is a valuable tool for guiding healthcare interventions toward those most likely to benefit. Yet, evaluating the pairing of a risk prediction model with an intervention using randomized controlled trials presents substantial challenges, making quasi-experimental designs an attractive alternatives. Existing designs, however, assume that both the model and the decision rules used to trigger interventions (typically a risk threshold) remain fixed. This limits their utility in modern healthcare, where both are routinely updated. We introduce a regression discontinuity framework that accommodates adaptation in both the model and the risk threshold. We precisely characterize the form of interference introduced by these adaptations and exploit this structure to establish conditions for identification and thus design estimation strategies. The key idea is to define counterfactual risks-the scores patients would have received under hypothetical reorderings-thereby restoring local exchangeability and enabling valid estimation of the local average treatment effect. Our estimator leverages the fact that, although counterfactual risk vectors grow with time, they typically lie in a low-dimensional space. In simulations of cardiovascular prevention programs, we show that our method accurately recovers treatment effects even as thresholds adapt to meet operational or clinical targets and models are updated to align predicted and observed outcomes or to exclude demographic predictors such as race.

• Publication year

  2025

• Venue

  Unknown venue

• Publication date

  2025-10-29

• Fields of study

  Medicine, Mathematics

• Identifiers
  arXiv 2510.25052
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• A scoping review of reporting gaps in FDA-approved AI medical devices

  2024cited by this paper
• Dashboarding to Monitor Machine-Learning-Based Clinical Decision Support Interventions

  2023cited by this paper
• There is no such thing as a validated prediction model

  2023cited by this paper
• Methodological guidance for the evaluation and updating of clinical prediction models: a systematic review

  2022cited by this paper
• Developing and Implementing Predictive Models in a Learning Healthcare System: Traditional and Artificial Intelligence Approaches in the Veterans Health Administration.

  2022influential reference
• Development and implementation experience of a learning healthcare system for facility based newborn care in low resource settings: The Neotree

  2022cited by this paper
• National Health and Nutrition Examination Survey 2017-March 2020 Prepandemic Data Files-Development of Files and Prevalence Estimates for Selected Health Outcomes.

  2021cited by this paper
• Collaborative design and implementation of a clinical decision support system for automated fall-risk identification and referrals in emergency departments.

  2021cited by this paper
• The Regression Discontinuity Design Under Interference: A Local Randomization-based Approach

  2021cited by this paper
• A Unifying Approach for GFR Estimation: Recommendations of the NKF-ASN Task Force on Reassessing the Inclusion of Race in Diagnosing Kidney Disease.

  2021cited by this paper
• Developing a delivery science for artificial intelligence in healthcare

  2020cited by this paper
• Artificial Intelligence in Clinical Decision Support: Challenges for Evaluating AI and Practical Implications

  2019cited by this paper
• Integrating Predictive Modeling Into Mental Health Care: An Example in Suicide Prevention.

  2019cited by this paper
• The Needs and Benefits of Continuous Model Updates on the Accuracy of RT-Induced Toxicity Prediction Models Within a Learning Health System.

  2019cited by this paper
• PROBAST: A Tool to Assess the Risk of Bias and Applicability of Prediction Model Studies

  2019cited by this paper
• A Potential Outcomes Calculus for Identifying Conditional Path-Specific Effects

  2019cited by this paper
• Enhanced American College of Cardiology/American Heart Association Strategy for Prevention of Sudden Cardiac Death in High-Risk Patients With Hypertrophic Cardiomyopathy.

  2019cited by this paper
• Training and Interpreting Machine Learning Algorithms to Evaluate Fall Risk After Emergency Department Visits

  2019cited by this paper
• Creating a Learning Health System through Rapid-Cycle, Randomized Testing.

  2019cited by this paper
• Key challenges for delivering clinical impact with artificial intelligence

  2019cited by this paper
• The Proliferation of Reports on Clinical Scoring Systems: Issues About Uptake and Clinical Utility.

  2019cited by this paper
• From development to deployment: dataset shift, causality, and shift-stable models in health AI.

  2019cited by this paper
• A review of statistical updating methods for clinical prediction models

  2018cited by this paper
• Using the Hendrich II Inpatient Fall Risk Screen to Predict Outpatient Falls After Emergency Department Visits

  2018cited by this paper
• Using Chief Complaint in Addition to Diagnosis Codes to Identify Falls in the Emergency Department

  2017cited by this paper
• International External Validation Study of the 2014 European Society of Cardiology Guidelines on Sudden Cardiac Death Prevention in Hypertrophic Cardiomyopathy (EVIDENCE-HCM)

  2017cited by this paper
• An observational study of how clinicians use cardiovascular risk assessment to inform statin prescribing decisions.

  2017cited by this paper
• Inference in Regression Discontinuity Designs under Local Randomization

  2016cited by this paper
• Interpreting Regression Discontinuity Designs with Multiple Cutoffs

  2016cited by this paper
• Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD)

  2015cited by this paper
• Predictive Modeling and Concentration of the Risk of Suicide: Implications for Preventive Interventions in the US Department of Veterans Affairs.

  2015cited by this paper
• Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): Explanation and Elaboration

  2015cited by this paper
• Evaluation of a New Zealand program to improve transition of care for older high risk adults

  2015cited by this paper
• 2013 ACC/AHA Guideline on the Assessment of Cardiovascular Risk: A Report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines

  2014influential reference
• A novel clinical risk prediction model for sudden cardiac death in hypertrophic cardiomyopathy (HCM risk-SCD).

  2014cited by this paper
• Single World Intervention Graphs ( SWIGs ) : A Unification of the Counterfactual and Graphical Approaches to Causality

  2013influential reference
• Implementing the Learning Health System: From Concept to Action

  2012cited by this paper
• How to Obtain NNT from Cohen's d: Comparison of Two Methods

  2011cited by this paper
• The oral glucose tolerance test (OGTT) revisited.

  2011cited by this paper
• Refining clinical risk stratification for predicting stroke and thromboembolism in atrial fibrillation using a novel risk factor-based approach: the euro heart survey on atrial fibrillation.

  2010cited by this paper
• EAU guidelines on renal cell carcinoma: 2014 update.

  2010cited by this paper
• Achieving a Nationwide Learning Health System

  2010cited by this paper
• A regression discontinuity design analysis of the incumbency advantage and tenure in the U.S. House

  2009cited by this paper
• What is Fair Participant Selection

  2009cited by this paper
• The perverse effects of partial employment protection reform: The case of French older workers

  2008cited by this paper
• Toward Causal Inference With Interference

  2008cited by this paper
• FRAX™ and the assessment of fracture probability in men and women from the UK

  2008cited by this paper
• Updating methods improved the performance of a clinical prediction model in new patients.

  2008cited by this paper
• The use of clinical risk factors enhances the performance of BMD in the prediction of hip and osteoporotic fractures in men and women

  2007cited by this paper
• Interference Between Units in Randomized Experiments

  2007cited by this paper
• Translating Clinical Research into Clinical Practice: Impact of Using Prediction Rules To Make Decisions

  2006cited by this paper
• Validation and extension of the Memorial Sloan-Kettering prognostic factors model for survival in patients with previously untreated metastatic renal cell carcinoma.

  2005cited by this paper
• Cross-sectional evaluation of the Finnish Diabetes Risk Score: a tool to identify undetected type 2 diabetes, abnormal glucose tolerance and metabolic syndrome

  2005cited by this paper
• Randomized experiments from non-random selection in U.S. House elections

  2005cited by this paper
• Validation and updating of predictive logistic regression models: a study on sample size and shrinkage

  2004cited by this paper
• A meta-analysis of previous fracture and subsequent fracture risk.

  2004cited by this paper
• Prediction of progression after radical nephrectomy for patients with clear cell renal cell carcinoma

  2003cited by this paper
• An outcome prediction model for patients with clear cell renal cell carcinoma treated with radical nephrectomy based on tumor stage, size, grade and necrosis: the SSIGN score.

  2002cited by this paper
• IDENTIFICATION AND ESTIMATION OF TREATMENT EFFECTS WITH A REGRESSION-DISCONTINUITY DESIGN

  2001cited by this paper
• Survival and prognostic stratification of 670 patients with advanced renal cell carcinoma.

  1999cited by this paper
• Wanted. Single, white male for medical research.

  1992cited by this paper
• Statistics and Causal Inference: Comment: Which Ifs Have Causal Answers

  1986cited by this paper
• Randomization Analysis of Experimental Data: The Fisher Randomization Test Comment

  1980cited by this paper
• Regression-Discontinuity Analysis: An Alternative to the Ex-Post Facto Experiment

  1960cited by this paper

• No citing papers are available for this paper.