Association of Clinician Diagnostic Performance With Machine Learning–Based Decision Support Systems

Key Points Question Is clinician diagnostic performance associated with the use of machine learning–based clinical decision support systems? Findings In this systematic review of 37 studies, no robust evidence was found to suggest an association between the use of machine learning–based clinical algorithms to support rather than replace human decision-making and improved clinician diagnostic performance. Meaning Caution should be observed when estimating the current ability of machine learning algorithms to affect patient care, and emphasis on the evaluation of the human-computer interaction is needed.

• Publication year

  2021

• Venue

  JAMA Network Open

• Publication date

  2021-03-01

• Fields of study

  Medicine, Computer Science

• Identifiers
  DOI 10.1001/jamanetworkopen.2021.1276PMID 33704476PMCID 7953308
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar, PubMed

• No claims are published for this paper.

• No concepts are published for this paper.

• Artificial Intelligence in Medicine: 23rd International Conference, AIME 2025, Pavia, Italy, June 23–26, 2025, Proceedings, Part I

  2025cited by this paper
• Guidelines for clinical trial protocols for interventions involving artificial intelligence: the SPIRIT-AI extension

  2020cited by this paper
• Augment Intelligence Dermatology : Deep Neural Networks Empower Medical Professionals in Diagnosing Skin Cancer and Predicting Treatment Options for 134 Skin Disorders.

  2020cited by this paper
• Artificial intelligence versus clinicians: systematic review of design, reporting standards, and claims of deep learning studies

  2020cited by this paper
• Detection of Breast Cancer with Mammography: Effect of an Artificial Intelligence Support System.

  2019cited by this paper
• A computer-aided diagnosis system using artificial intelligence for the diagnosis and characterization of breast masses on ultrasound

  2019cited by this paper
• High-performance medicine: the convergence of human and artificial intelligence

  2019cited by this paper
• A comparison of deep learning performance against health-care professionals in detecting diseases from medical imaging: a systematic review and meta-analysis.

  2019cited by this paper
• Reporting of artificial intelligence prediction models.

  2019cited by this paper
• Effect of a Deep Learning Framework-Based Computer-Aided Diagnosis System on the Diagnostic Performance of Radiologists in Differentiating between Malignant and Benign Masses on Breast Ultrasonography

  2019cited by this paper
• Improved Cancer Detection Using Artificial Intelligence: a Retrospective Evaluation of Missed Cancers on Mammography

  2019cited by this paper
• Development and Validation of a Deep Learning–Based Automated Detection Algorithm for Major Thoracic Diseases on Chest Radiographs

  2019cited by this paper
• Diagnostic Accuracy of CT for Prediction of Bladder Cancer Treatment Response with and without Computerized Decision Support.

  2019cited by this paper
• Using a Deep Learning Algorithm and Integrated Gradients Explanation to Assist Grading for Diabetic Retinopathy.

  2019cited by this paper
• Deep neural network improves fracture detection by clinicians

  2018cited by this paper
• Performance and clinical impact of machine learning based lung nodule detection using vessel suppression in melanoma patients.

  2018cited by this paper
• A cloud-based computer-aided detection system improves identification of lung nodules on computed tomography scans of patients with extra-thoracic malignancies

  2018cited by this paper
• Focal breast lesion characterization according to the BI-RADS US lexicon: role of a computer-aided decision-making support

  2018cited by this paper
• Effects of computerized decision support system implementations on patient outcomes in inpatient care: a systematic review

  2018cited by this paper
• Application of Computer‐Aided Diagnosis on Breast Ultrasonography: Evaluation of Diagnostic Performances and Agreement of Radiologists According to Different Levels of Experience

  2018cited by this paper
• Computer-aided diagnosis for (123I)FP-CIT imaging: impact on clinical reporting

  2018cited by this paper
• Deep-learning-assisted diagnosis for knee magnetic resonance imaging: Development and retrospective validation of MRNet

  2018cited by this paper
• Impact of Data Presentation on Physician Performance Utilizing Artificial Intelligence-Based Computer-Aided Diagnosis and Decision Support Systems

  2018cited by this paper
• Impact of Deep Learning Assistance on the Histopathologic Review of Lymph Nodes for Metastatic Breast Cancer

  2018cited by this paper
• Multiparametric magnetic resonance imaging of the prostate with computer-aided detection: experienced observer performance study

  2017cited by this paper
• Application of computer-aided diagnosis in breast ultrasound interpretation: improvements in diagnostic performance according to reader experience

  2017cited by this paper
• Computer-aided detection of brain metastasis on 3D MR imaging: Observer performance study

  2017cited by this paper
• ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions

  2016cited by this paper
• Application of Computer-Aided Diagnosis to the Sonographic Evaluation of Cervical Lymph Nodes

  2016cited by this paper
• CADBOSS: A computer-aided diagnosis system for whole-body bone scintigraphy scans.

  2016cited by this paper
• The Effectiveness of Electronic Differential Diagnoses (DDX) Generators: A Systematic Review and Meta-Analysis

  2016influential reference
• Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): The TRIPOD Statement

  2015cited by this paper
• Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation

  2015influential reference
• Diagnostic Accuracy of Digital Screening Mammography With and Without Computer-Aided Detection.

  2015cited by this paper
• Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement

  2015cited by this paper
• Impact of computer-aided detection systems on radiologist accuracy with digital mammography.

  2014cited by this paper
• A Computer‐Aided Diagnostic Algorithm Improves the Accuracy of Transesophageal Echocardiography for Left Atrial Thrombi

  2014cited by this paper
• Single reading with computer-aided detection performed by selected radiologists in a breast cancer screening program.

  2014cited by this paper
• 診断精度研究のバイアスリスク評価ツールQUADAS－2：a Revised Tool for the Quality Assessment of Diagnostic Accuracy Studies 2 の活用

  2014cited by this paper
• Short-Term Outcomes of Screening Mammography Using Computer-Aided Detection

  2013cited by this paper
• Evaluation of the accuracy of a computer-aided diagnosis (CAD) system in breast ultrasound according to the radiologist's experience.

  2012cited by this paper
• Effect of Clinical Decision-Support Systems

  2012cited by this paper
• Content-based image-retrieval system in chest computed tomography for a solitary pulmonary nodule: method and preliminary experiments

  2012cited by this paper
• Impact on breast cancer diagnosis in a multidisciplinary unit after the incorporation of mammography digitalization and computer-aided detection systems.

  2011cited by this paper
• Impact of a CAD system in a screen-film mammography screening program: a prospective study.

  2011cited by this paper
• Effectiveness of computer-aided detection in community mammography practice.

  2011cited by this paper
• Effects of clinical decision-support systems on practitioner performance and patient outcomes: a synthesis of high-quality systematic review findings

  2011cited by this paper
• Computer-aided detection of pulmonary embolism at CT pulmonary angiography: can it improve performance of inexperienced readers?

  2011cited by this paper
• QUADAS-2: A Revised Tool for the Quality Assessment of Diagnostic Accuracy Studies

  2011cited by this paper
• Evaluation of clinical breast MR imaging performed with prototype computer-aided diagnosis breast MR imaging workstation: reader study.

  2011cited by this paper
• An image feature approach for computer-aided detection of ischemic stroke

  2011cited by this paper
• Computer-aided diagnosis of lung nodules on CT scans: ROC study of its effect on radiologists' performance.

  2010cited by this paper
• CAD in full-field digital mammography-influence of reader experience and application of CAD on interpretation of time.

  2010cited by this paper
• Using computer-aided detection in mammography as a decision support

  2010cited by this paper
• Does computer-assisted detection of pulmonary emboli enhance severity assessment and risk stratification in acute pulmonary embolism?

  2010cited by this paper
• Computer-aided detection in full-field digital mammography in a clinical population: performance of radiologist and technologists

  2010cited by this paper
• Preferred reporting items for systematic reviews and meta-analyses: the PRISMA Statement

  2009cited by this paper
• Human Factors in the Design and Evaluation of Central Control Room Operations

  2009cited by this paper
• Human factors approach for evaluation and redesign of human-system interfaces of a nuclear power plant simulator

  2008cited by this paper
• Human factors issues in implementation of advanced aviation technologies: a case of false alerts and cockpit displays of traffic information

  2006cited by this paper
• Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review.

  2005cited by this paper
• Using SHERPA to predict design-induced error on the flight deck

  2005cited by this paper
• Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review.

  2005cited by this paper
• The Importance of Cognitive Errors in Diagnosis and Strategies to Minimize Them

  2003cited by this paper
• Human error in European air traffic management: the HERA project

  2002cited by this paper
• Some Studies in Machine Learning Using the Game of Checkers

  1995cited by this paper
• Anything you can do, I can do better (No you can't)

  1986cited by this paper
• GrADE: an emerging consensus on rating quality of evidence and strength of recommendations

  year unknowncited by this paper
• European Journal of Radiology Open

  year unknowncited by this paper

• When artificial intelligence guides and misguides clinicians: A critical appraisal of AI recommendation correctness and diagnostic decision-making

  2026cites this paper
• Evaluating Generative AI (Microsoft Copilot) as an Adjunctive Decision-Support System in Oral and Maxillofacial Radiology: A Retrospective Study

  2026cites this paper
• Artificial Intelligence Payment Policies: Challenges For CMS And The Medicare Physician Fee Schedule.

  2026cites this paper
• Improved YOLOv8 with average pooling downsampling for detection and classification of intertrochanteric femoral fractures in X-ray images: a study focusing on AO/OTA classification

  2026cites this paper
• Advances in Photodynamic Therapy: Photosensitizers, Biological Mechanisms, and Artificial Intelligence-Driven Innovation

  2026cites this paper
• Characterizing bacteremia in neonatal calves with diarrhea: a case-control study.

  2026cites this paper
• Value of Machine Learning Models for Cell-Free DNA-Based Multi-Cancer Early Detection: A Systematic Review and Meta-Analysis.

  2026cites this paper
• Accuracy of ultrasound for intussusception in pediatric emergency presentations: a systematic review and diagnostic meta-analysis

  2026cites this paper
• Performance of Large Language Models for Radiology Report Impression Generation: A Systematic Review

  2026cites this paper
• Clinical research on artificial intelligence medical diagnostic devices: A scoping review

  2026cites this paper
• Predicting the likelihood of hypoadrenocorticism in dogs using signalment and routine laboratory results with an ensemble machine learning predictive model.

  2026cites this paper
• Artificial Intelligence in Clinical Medicine: Challenges Across Diagnostic Imaging, Clinical Decision Support, Surgery, Pathology, and Drug Discovery

  2025influential citation
• Artificial intelligence and medical diagnosis: past, present and future

  2025cites this paper
• Translationale Herausforderungen und klinisches Potenzial von künstlicher Intelligenz in der minimal-invasiven Chirurgie

  2025cites this paper
• Medical laboratory data-based models: opportunities, obstacles, and solutions

  2025cites this paper
• Evaluation of a Deep Learning and XAI based Facial Phenotyping Tool for Genetic Syndromes: A Clinical User Study

  2025influential citation
• Using ChatGPT in ethical dilemmas and policy-related complex decision making: Are we ready yet?

  2025cites this paper
• How to bridge the gap between evidence-based and clinical management of endometriosis

  2025cites this paper
• Regulating the future of laboratory medicine: European regulatory landscape of AI-driven medical device software in laboratory medicine

  2025cites this paper
• To Recommend or Not to Recommend: Designing and Evaluating AI-Enabled Decision Support for Time-Critical Medical Events

  2025cites this paper
• Applications of machine learning and deep learning in musculoskeletal medicine: a narrative review

  2025cites this paper
• Artificial Intelligence in Biliopancreatic Disorders: Applications in Cross-Imaging and Endoscopy.

  2025cites this paper
• All in the Name of Artificial Intelligence: A Commentary on Linardon (2025)

  2025cites this paper
• Prospective multicenter external validation of the rib fracture frailty index.

  2025cites this paper
• Utilizing large language models for gastroenterology research: a conceptual framework

  2025cites this paper
• The case for delegated AI autonomy for Human AI teaming in healthcare

  2025cites this paper
• Performance of Commercial Deep Learning-Based Auto-Segmentation Software for Prostate Cancer Radiation Therapy Planning: A Systematic Review

  2025cites this paper
• Personalized dose selection platform for patients with solid tumors in the PRECISE CURATE.AI feasibility trial

  2025cites this paper
• Deployable machine learning-based decision support system for tracheostomy in acute burn patients

  2025cites this paper
• AI-driven early detection of severe influenza in Jiangsu, China: a deep learning model validated through the design of multi-center clinical trials and prospective real-world deployment

  2025cites this paper
• Artificial intelligence–based clinical decision support in the emergency department: A scoping review

  2025influential citation
• CORE-MD clinical risk score for regulatory evaluation of artificial intelligence-based medical device software

  2025cites this paper
• Impact of AI recommendation correctness on diagnostic accuracy in clinical decision-making

  2025cites this paper
• Identification of Key Factors and Symmetrical Hierarchical Paths Influencing the Efficiency of Medical Human–Machine Collaborative Diagnosis Based on DEMATEL-ISM

  2025cites this paper
• Comparative Evaluation of Proprietary and Open-Source Large Language Models for Systematic Multi-source Information Extraction in Interventional Oncology.

  2025cites this paper
• Human-AI teaming in healthcare: 1 + 1 > 2?

  2025cites this paper
• Beyond amplification: how artificial intelligence is redefining hearing

  2025cites this paper
• Contrastive text embeddings with effective sample mining for enhanced disease diagnosis

  2025cites this paper
• Transformer-based Clinical Decision Support Systems using Structured and Unstructured EHR Data

  2025cites this paper
• Task-Technology Fit of Artificial Intelligence-based clinical decision support systems: a review of qualitative studies

  2025cites this paper
• Evaluating the impact of AI assistance on decision-making in emergency doctors interpreting chest X-rays: a multi-reader multi-case study

  2025cites this paper
• Artificial Intelligence in Prostate MRI: Current Evidence and Clinical Translation Challenges—A Narrative Review

  2025influential citation
• Artificial Intelligence in Malnutrition: A Systematic Literature Review

  2024cites this paper
• Enabling Collaborative Clinical Diagnosis of Infectious Keratitis by Integrating Expert Knowledge and Interpretable Data-driven Intelligence

  2024cites this paper
• Factors influencing clinician and patient interaction with machine learning-based risk prediction models: a systematic review.

  2024cites this paper
• Diagnosis and prognosis of abnormal cardiac scintigraphy uptake suggestive of cardiac amyloidosis using artificial intelligence: a retrospective, international, multicentre, cross-tracer development and validation study.

  2024cites this paper
• Factors Associated with Caregiver Decision Not to Transport Pediatric Patients Assessed by Emergency Medical Services

  2024cites this paper
• Advancing Bloodstream Infection Prediction Using Explanable Artificial Intelligence Framework

  2024cites this paper
• A Justifiable Investment in AI for Healthcare: Aligning Ambition with Reality

  2024cites this paper
• Predictors of Health Care Practitioners’ Intention to Use AI-Enabled Clinical Decision Support Systems: Meta-Analysis Based on the Unified Theory of Acceptance and Use of Technology

  2024cites this paper
• Effect of Machine Learning on Anaesthesiology Clinician Prediction of Postoperative Complications: The Perioperative ORACLE Randomised Clinical Trial

  2024cites this paper
• Pneumonia diagnosis performance in the emergency department: a mixed-methods study about clinicians’ experiences and exploration of individual differences and response to diagnostic performance feedback

  2024cites this paper
• Predicting Chronic Pain and Treatment Outcomes Using Machine Learning Models Based on High-Dimensional Clinical Data From a Large Retrospective Cohort.

  2024cites this paper
• Artificial intelligence for breast cancer: Implications for diagnosis and management

  2024cites this paper
• Development and translation of human-AI interaction models into working prototypes for clinical decision-making

  2024cites this paper
• Training and assessing convolutional neural network performance in automatic vascular segmentation using Ga-68 DOTATATE PET/CT

  2024cites this paper
• Beyond Step Count: Are We Ready to Use Digital Phenotyping to Make Actionable Individual Predictions in Psychiatry?

  2024cites this paper
• Explainable AI in Healthcare: Systematic Review of Clinical Decision Support Systems

  2024cites this paper
• Artificial intelligence for hearing loss prevention, diagnosis, and management

  2024cites this paper
• Patient-Oriented Questionnaires and Machine Learning for Rare Disease Diagnosis: A Systematic Review

  2024cites this paper
• The Accuracy of Artificial Intelligence Models in Hand/Wrist Fracture and Dislocation Diagnosis

  2024cites this paper
• Effect of machine learning models on clinician prediction of postoperative complications: the Perioperative ORACLE randomised clinical trial.

  2024cites this paper
• Towards Accountable AI in Eye Disease Diagnosis: Workflow, External Validation, and Development

  2024cites this paper
• Systemic Metabolic and Volumetric Assessment via Whole-Body [18F]FDG-PET/CT: Pancreas Size Predicts Cachexia in Head and Neck Squamous Cell Carcinoma

  2024cites this paper
• Post-Exercise Recovery Score Using Computational Cardiac Model

  2024cites this paper
• Evaluating Machine Learning Models for Stroke Prognosis and Prediction in Atrial Fibrillation Patients: A Comprehensive Meta-Analysis

  2024cites this paper
• Leveraging explainable artificial intelligence for early prediction of bloodstream infections using historical electronic health records

  2024cites this paper
• Performance of Commercial Deep Learning-Based Auto-Segmentation Software for Breast Cancer Radiation Therapy Planning: A Systematic Review

  2024cites this paper
• Machine Learning and Deep Learning for Diagnosis of Lumbar Spinal Stenosis: Systematic Review and Meta-Analysis

  2024cites this paper
• Investigating Clinicians’ Intentions and Influencing Factors for Using an Intelligence-Enabled Diagnostic Clinical Decision Support System in Health Care Systems: Cross-Sectional Survey

  2024cites this paper
• Effects of machine learning-based clinical decision support systems on decision-making, care delivery, and patient outcomes: a scoping review

  2023cites this paper
• A Counterblast to Pessimists and Naysayers – Intelligent Echocardiography Remains the Foundation Stone of Evidence-Based Clinical Cardiology

  2023cites this paper
• The Accuracy and Potential Racial and Ethnic Biases of GPT-4 in the Diagnosis and Triage of Health Conditions: Evaluation Study

  2023cites this paper
• FDA Regulations of AI-Driven Clinical Decision Support Devices Fall Short.

  2023cites this paper
• The Implementation of AI in Clinical Decision Support System: Effects on Patient Outcomes and Operational Costs

  2023cites this paper
• Applying AI and Guidelines to Assist Medical Students in Recognizing Patients With Heart Failure: Protocol for a Randomized Trial

  2023cites this paper
• Translation of AI into oncology clinical practice

  2023cites this paper
• Development of an artificial intelligence bacteremia prediction model and evaluation of its impact on physician predictions focusing on uncertainty

  2023cites this paper
• Generative Adversarial Network (Generative Artificial Intelligence) in Pediatric Radiology: A Systematic Review

  2023cites this paper
• Methods used to evaluate usability of mobile clinical decision support systems for healthcare emergencies: a systematic review and qualitative synthesis

  2023cites this paper
• Integration of Artificial Intelligence Decision Aids to Reduce Workload and Enhance Efficiency in Thyroid Nodule Management

  2023cites this paper
• Artificial intelligence in medical device software and high-risk medical devices – a review of definitions, expert recommendations and regulatory initiatives

  2023cites this paper
• Online Decision Mediation

  2023cites this paper
• Clinical applications of artificial intelligence in radiology

  2023cites this paper
• EQRbot: A chatbot delivering EQR argument-based explanations

  2023cites this paper
• Artificial Intelligence for Hip Fracture Detection and Outcome Prediction

  2023cites this paper
• Diagnostic Performance of Artificial Intelligence-Based Computer-Aided Detection and Diagnosis in Pediatric Radiology: A Systematic Review

  2023influential citation
• The Steep Road to Artificial Intelligence-mediated Radiology.

  2023cites this paper
• Digital health literacy to enhance workforce skills and clinical effectiveness: A response to ‘Digital health literacy: Helpful today, dependency tomorrow? Contingency planning in a digital age’

  2023cites this paper
• Harnessing the power of clinical decision support systems: challenges and opportunities

  2023cites this paper
• Machine learning-based clinical decision support using laboratory data

  2023cites this paper
• The Need for the Human-Centred Explanation for ML-based Clinical Decision Support Systems

  2023cites this paper
• Holistic Human-Serving Digitization of Health Care Needs Integrated Automated System-Level Assessment Tools

  2023cites this paper
• Machine Learning Techniques Applied to Heart Disease Diagnosis System

  2023cites this paper
• Imaging Artificial Intelligence: A Framework for Radiologists to Address Health Equity, From the AJR Special Series on DEI.

  2023cites this paper
• Computer-assisted analysis of routine electroencephalogram to identify hidden biomarkers of epilepsy: protocol for a systematic review

  2023cites this paper
• DECIDE-AI: a new reporting guideline and its relevance to artificial intelligence studies in radiology.

  2023cites this paper
• Using information technology to reduce diagnostic error: still a bridge too far?

  2022cites this paper
• Clinical Informatics and Quality Improvement in the Pediatric Intensive Care Unit.

  2022cites this paper
• Potentials and prospects for digital medical ecosystems

  2022cites this paper