The use of receiver operating characteristic curves in biomedical informatics

Receiver operating characteristic (ROC) curves are frequently used in biomedical informatics research to evaluate classification and prediction models for decision support, diagnosis, and prognosis. ROC analysis investigates the accuracy of a model's ability to separate positive from negative cases (such as predicting the presence or absence of disease), and the results are independent of the prevalence of positive cases in the study population. It is especially useful in evaluating predictive models or other tests that produce output values over a continuous range, since it captures the trade-off between sensitivity and specificity over that range. There are many ways to conduct an ROC analysis. The best approach depends on the experiment; an inappropriate approach can easily lead to incorrect conclusions. In this article, we review the basic concepts of ROC analysis, illustrate their use with sample calculations, make recommendations drawn from the literature, and list readily available software.

• Publication year

  2005

• Venue

  Journal of Biomedical Informatics

• Publication date

  2005-10-01

• Fields of study

  Medicine, Computer Science

• Identifiers
  DOI 10.1016/J.JBI.2005.02.008PMID 16198999
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar, PubMed

• No claims are published for this paper.

• No concepts are published for this paper.

• The Statistical Evaluation of Medical Tests for Classification and Prediction

  2005influential reference
• Research Paper: Text Categorization Models for High-Quality Article Retrieval in Internal Medicine

  2004cited by this paper
• Sources of Variation and Bias in Studies of Diagnostic Accuracy

  2004cited by this paper
• Data mining techniques for cancer detection using serum proteomic profiling

  2004cited by this paper
• support vector

  2004cited by this paper
• ROC curves in clinical chemistry: uses, misuses, and possible solutions.

  2004cited by this paper
• ROC methodology within a monitoring framework

  2003cited by this paper
• Receiver operating characteristic curves and their use in radiology.

  2003cited by this paper
• Research Paper: Creating a Text Classifier to Detect Radiology Reports Describing Mediastinal Findings Associated with Inhalational Anthrax and Other Disorders

  2003cited by this paper
• Preoperative prediction of malignancy of ovarian tumors using least squares support vector machines

  2003cited by this paper
• Determining sample size for ROC studies: what is reasonable for the expected difference in tests' ROC areas?

  2003cited by this paper
• Research Paper: Evaluation of a Cardiac Diagnostic Program in a Typical Clinical Setting

  2003cited by this paper
• Comparison of eight computer programs for receiver-operating characteristic analysis.

  2003cited by this paper
• Confidence bounds when the estimated ROC area is 1.01.

  2002cited by this paper
• A non‐parametric method for the comparison of partial areas under ROC curves and its application to large health care data sets

  2002cited by this paper
• Comparison of three methods for estimating the standard error of the area under the curve in ROC analysis of quantitative data.

  2002influential reference
• Statistical Methods in Diagnostic Medicine

  2002cited by this paper
• Estimation of the area under the ROC curve

  2002influential reference
• METHODOLOGICAL REVIEW Wavelets and Imaging Informatics: A Review of the Literature

  2001cited by this paper
• A Comparison of Classification Algorithms to Automatically Identify Chest X-Ray Reports That Support Pneumonia

  2001cited by this paper
• A Simple Generalisation of the Area Under the ROC Curve for Multiple Class Classification Problems

  2001cited by this paper
• Meta-analysis of ROC Curves

  2000cited by this paper
• Comparing Three-class Diagnostic Tests by Three-way ROC Analysis

  2000cited by this paper
• Three-way ROCs

  1999cited by this paper
• The interpretation of diagnostic tests

  1999cited by this paper
• Confidence intervals for the receiver operating characteristic area in studies with small samples.

  1998influential reference
• Maximum likelihood estimation of receiver operating characteristic (ROC) curves from continuously-distributed data.

  1998cited by this paper
• Statistical Comparison of Two ROC-curve Estimates Obtained from Partially-paired Datasets

  1998cited by this paper
• Using Smoothed Receiver Operating Characteristic Curves to Summarize and Compare Diagnostic Systems

  1998cited by this paper
• Nonparametric analysis of clustered ROC curve data.

  1997cited by this paper
• Sampling variability of nonparametric estimates of the areas under receiver operating characteristic curves: an update.

  1997influential reference
• A Comparison of Parametric and Nonparametric Approaches to ROC Analysis of Quantitative Diagnostic Tests

  1997cited by this paper
• Smooth non-parametric receiver operating characteristic (ROC) curves for continuous diagnostic tests.

  1997cited by this paper
• A distribution-free procedure for comparing receiver operating characteristic curves from a paired experiment

  1996cited by this paper
• MedCalc: a new computer program for medical statistics.

  1995cited by this paper
• Software for illustrative presentation of basic clinical characteristics of laboratory tests--GraphROC for Windows.

  1995cited by this paper
• Computing Sample Size for Receiver Operating Characteristic Studies

  1994cited by this paper
• A computer program for non-parametric receiver operating characteristic analysis.

  1993cited by this paper
• Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine.

  1993cited by this paper
• Receiver operating characteristic rating analysis. Generalization to the population of readers and patients with the jackknife method.

  1992cited by this paper
• Analyzing a Portion of the ROC Curve

  1989cited by this paper
• Some practical issues of experimental design and data analysis in radiological ROC studies.

  1989cited by this paper
• A family of nonparametric statistics for comparing diagnostic markers with paired or unpaired data

  1989cited by this paper
• Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach.

  1988influential reference
• Measuring the accuracy of diagnostic systems.

  1988cited by this paper
• The interpretation of diagnostic tests.

  1987cited by this paper
• ROC Methodology in Radiologic Imaging

  1986cited by this paper
• A method of comparing the areas under receiver operating characteristic curves derived from the same cases.

  1983cited by this paper
• The meaning and use of the area under a receiver operating characteristic (ROC) curve.

  1982influential reference
• Evaluating the yield of medical tests.

  1982cited by this paper
• The area above the ordinal dominance graph and the area below the receiver operating characteristic graph

  1975cited by this paper

• MARS modelling for spatial analysis of coastal erosion susceptibility

  2026cites this paper
• Exploring the determinants of non-suicidal self-injury among university students in Jordan: a cross-sectional study

  2026cites this paper
• Nasopharyngeal cancer screening and immunotherapy efficacy evaluation based on plasma separation combined with label-free SERS technology.

  2025cites this paper
• Radiomic study of common sellar region lesions differentiation in magnetic resonance imaging based on multi-classification machine learning model

  2025cites this paper
• Development and validation of a machine learning model for early screening of high-risk mild cognitive impairment from the multi-cohort data

  2025cites this paper
• Ultrasensitive CSF rhinorrhea screening via machine learning-aided SERS on Au@Ag nanopillars

  2025cites this paper
• FindMeIfYouCan: Bringing Open Set metrics to near, far and farther Out-of-Distribution Object Detection

  2025cites this paper
• Integrated in silico and in vitro approaches identify SNX.2112 as a drug vulnerability in t(7;12) AML stem-like cells

  2025cites this paper
• Estimation of Diagnostic Test Accuracy Without Gold Standards

  2025cites this paper
• Forecast of cytotoxic T lymphocyte epitope using sequence weighting and artificial neural network based on EasyPred modeler

  2025cites this paper
• Test-Retest Reliability and Responsiveness for the Simplified Chinese Lower Extremity Functional Scale in Patients With Lower Extremity Musculoskeletal Injuries

  2025cites this paper
• Performance of PCA3 and TMPRSS2:ERG Within the Prostate Cancer Prevention Trial Risk Calculator Version 2 in a Lithuanian Cohort

  2025influential citation
• Can circadian rhythms of heart rate variability identify major depressive disorder? - A study based on support vector machine analysis.

  2025cites this paper
• Defining region and duration-specific substantial clinical benefit values for the simplified Chinese lower extremity functional scale

  2025influential citation
• Leveraging AlphaFold2 structural space exploration for generating drug target structures in structure-based virtual screening

  2025cites this paper
• A comprehensive survey of imputation methods in medical missing data analysis

  2025cites this paper
• Diagnostic value of alpha-fetoprotein and prothrombin induced by vitamin K absence-II in serum, bile, and feces in hepatocellular carcinoma

  2025cites this paper
• Explainability enhanced liver disease diagnosis technique using tree selection and stacking ensemble-based random forest model

  2025cites this paper
• Analysis of Z-Score and total score of athleticism on drafted and undrafted players from the NFL scouting combine

  2025cites this paper
• Interpreting Quadriceps Strength Thresholds for Patients Following ACL-Reconstruction: A Clinical Commentary

  2025cites this paper
• GIS-based multi-criteria predictive modelling for geothermal energy exploration

  2025cites this paper
• What makes you entrepreneurial? Using machine learning to predict technology entrepreneurship

  2025cites this paper
• Threshold-Optimized and Calibrated Logistic Regression for Breast Cancer Classification

  2025cites this paper
• ROC analysis determines optimal cutoff and doubtable interval of the Bath ankylosing spondylitis metrology index in a multicenter case control study

  2025cites this paper
• Mesoscale air motion and thermodynamics predict heavy hourly U.S. precipitation

  2024cites this paper
• DSG-KD: Knowledge Distillation From Domain-Specific to General Language Models

  2024cites this paper
• Early prediction of long hospital stay for Intensive Care units readmission patients using medication information

  2024cites this paper
• Quantitative Definition of Low-Health-Interest Populations by Using Regression Trees: A Nationwide Internet Survey in Japan

  2024cites this paper
• Aging Impairs Unimanual and Bimanual Hand-Grip Force Control Capabilities

  2024cites this paper
• Development and Comparison of Multiple Emotion Classification Models in Indonesia Text Using Machine Learning

  2024cites this paper
• Patients Undergoing Hip Arthroscopy with Concomitant Periacetabular Osteotomy Demonstrate Clinically Meaningful Improvement at 2 years using the PROMIS and iHOT-12.

  2024cites this paper
• A multi‐omics study to monitor senescence‐associated secretory phenotypes of Alzheimer's disease

  2024cites this paper
• Classification of battery compounds using structure-free Mendeleev encodings

  2024cites this paper
• Analysis of Plasma-Derived Exosomal MicroRNAs as Potential Biomarkers for Canine Idiopathic Epilepsy

  2024cites this paper
• Highly Multiplexed Reverse-Transcription Loop-Mediated Isothermal Amplification and Nanopore Sequencing (LAMPore) for Wastewater-Based Surveillance

  2024cites this paper
• Predicting Outstanding Results Following Primary Total Hip Arthroplasty Using The Maximal Outcome Improvement Threshold.

  2024cites this paper
• Wind turbine condition monitoring based on three fitted performance curves

  2024cites this paper
• A comparison of conventional and resampled personal reliability in detecting careless responding

  2024influential citation
• Predicting positive Clostridioides difficile test results using large-scale longitudinal data of demographics and medication history

  2024cites this paper
• Adaption and National Validation of a Tool for Predicting Mortality from Other Causes Among Men with Nonmetastatic Prostate Cancer.

  2024cites this paper
• Comparison of deep learning schemes in grading non-alcoholic fatty liver disease using B-mode ultrasound hepatorenal window images with liver biopsy as the gold standard.

  2024cites this paper
• News Sentiment and Liquidity Risk Forecasting: Insights from Iranian Banks

  2024cites this paper
• Anthropogenic pressures and spatio-temporal dynamics of forest ecosystems in the rural and border municipality of Kasenga (DRC)

  2024cites this paper
• Mechanism-centric regulatory network identifies NME2 and MYC programs as markers of Enzalutamide resistance in CRPC

  2024cites this paper
• Evaluation of penalized and machine learning methods for asthma disease prediction in the Korean Genome and Epidemiology Study (KoGES)

  2024cites this paper
• Exploring the geomorphological adequacy of the landslide susceptibility maps: A test for different types of landslides in the Bidente river basin (northern Italy)

  2024cites this paper
• Statistical Inference for Box–Cox based Receiver Operating Characteristic Curves

  2024cites this paper
• Deep‐learning computer vision can identify increased nuchal translucency in the first trimester of pregnancy

  2024influential citation
• Early differential diagnosis of pancytopenia related diseases based on serum surface-enhanced Raman spectroscopy.

  2024cites this paper
• Evaluating Feature Selection Methods for Accurate Diagnosis of Diabetic Kidney Disease

  2024cites this paper
• A study on the performance of the probability-limit control charts based on the geometric and negative binomial distributions

  2023cites this paper
• Combining a Frailty Index Based on Laboratory Data and Pneumonia Severity Assessments to Predict In-Hospital Outcomes in Older Adults with Community-Acquired Pneumonia

  2023cites this paper
• O-GlyThr: Prediction of human O-linked threonine glycosites using multi-feature fusion.

  2023cites this paper
• Investigating the Effects of Cell Size in Statistical Landslide Susceptibility Modelling for Different Landslide Typologies: A Test in Central–Northern Sicily

  2023cites this paper
• Discovery of AcrAB-TolC pump inhibitors: Virtual screening and molecular dynamics simulation approach

  2023cites this paper
• Identifying Predictors of Extubation on the day of Passing an SBT in Critically ill Adults

  2023cites this paper
• Endoscopic mucosal ischemic index for predicting anastomotic complications after esophagectomy: a prospective cohort study

  2023cites this paper
• An assessment of multi-criteria predictive model for alluvial sapphire exploration in Belel and environs, Adamawa Yadé-domain, northern Cameroon

  2023cites this paper
• Efficiency assessment of the environmental variables in the Qanat systems based on major hydrological basins in Iran

  2023cites this paper
• Futuristic Structural and Lithological Constraint Mapping of Landslides Using Structural Geology and Geospatial Techniques

  2023cites this paper
• Nuclear receptor co-repressor NCOR2 and its relation to GPER with prognostic impact in ovarian cancer

  2023cites this paper
• Combining multi-typologies landslide susceptibility maps: a case study for the Visso area (central Italy)

  2023cites this paper
• DARVIC: Dihedral angle-reliant variant impact classifier for functional prediction of missense VUS

  2023cites this paper
• Identification of RNA-based cell-type markers for stem-cell manufacturing systems with a statistical scoring function

  2023cites this paper
• Defining Thresholds and Predictors for Achieving the Patient Acceptable Symptom State for Patient-Reported Outcome Measures After Revision Hip Arthroscopy

  2023cites this paper
• Model-agnostic unsupervised detection of bots in a Likert-type questionnaire

  2023cites this paper
• Machine learning in the detection of dental cyst, tumor, and abscess lesions

  2023cites this paper
• Validation of the social communication questionnaire amongst Nigerian adolescents

  2023cites this paper
• Early Detection of Lung Disease using Multi-Class Classifiers

  2023cites this paper
• Prognostic factors for critically ill surgical patients with unplanned intensive care unit readmission: Developing a novel predictive scoring model for predicting readmission.

  2023cites this paper
• Identification of efflux pump inhibitors for Pseudomonas aeruginosa MexAB-OprM via ligand-based pharmacophores, 2D-QSAR, molecular docking, and molecular dynamics approaches

  2023cites this paper
• Detection of nasopharyngeal cancer cells using the laser tweezer Raman spectroscopy technology.

  2023cites this paper
• Approaches for Benchmarking Single-Cell Gene Regulatory Network Methods

  2023influential citation
• Mechanisms of cognitive disinhibition explain individual differences in adult attention deficit hyperactivity disorder traits.

  2023cites this paper
• Usability of an at-home tablet-based cognitive test in older adults with and without cognitive impairment

  2023influential citation
• Integration of High-Accuracy Geospatial Data and Machine Learning Approaches for Soil Erosion Susceptibility Mapping in the Mediterranean Region: A Case Study of the Macta Basin, Algeria

  2023cites this paper
• Improving the Efficacy of ACR TI-RADS Through Deep Learning-Based Descriptor Augmentation

  2023cites this paper
• Using Public Landslide Inventories for Landslide Susceptibility Assessment at the Basin Scale: Application to the Torto River Basin (Central-Northern Sicily, Italy)

  2023cites this paper
• Strawberry R-CNN: Recognition and counting model of strawberry based on improved faster R-CNN

  2023cites this paper
• Spatial and Temporal Analysis of Climatic Precursors before Major Earthquakes in Iran (2011–2021)

  2023cites this paper
• The effect of climate on the occurrence and abundance of tree recruitment in the province of Quebec, Canada

  2023cites this paper
• Proposition of In silico Pharmacophore Models for Malaria: A Review.

  2023cites this paper
• Serum Proteomic Biomarkers Diagnostic of Knee Osteoarthritis.

  2023cites this paper
• A Point-Based Risk Calculator for Mortality After Hepatectomy

  2023cites this paper
• Expert Algorithm for Substance Identification Using Mass Spectrometry: Statistical Foundations in Unimolecular Reaction Rate Theory

  2023cites this paper
• Comparing Factors Associated with Increased Stimulant Use in Relation to HIV Status Using a Machine Learning and Prediction Modeling Approach

  2023influential citation
• Comparison of the PROMIS and iHOT-12 in Determining Satisfaction Levels After Hip Arthroscopy for FAIS

  2023influential citation
• Expert Algorithm for Substance Identification Using Mass Spectrometry: Application to the Identification of Cocaine on Different Instruments Using Binary Classification Models

  2023cites this paper
• Correlation of finger-to-floor distance with the spinal mobility, spinal function indices and initial determination of its optimal cutoff value: a multicentre case–control study

  2023cites this paper
• BASICS OF RECEIVER OPERATING CHARACTERISTIC CURVES

  2023cites this paper
• Discovery of novel flavonoid derivatives as potential dual inhibitors against α-glucosidase and α-amylase: virtual screening, synthesis, and biological evaluation

  2023cites this paper
• Predicting COVID-19 severity using major risk factors and received vaccines

  2022cites this paper
• Landform classification: a high-performing mapping unit partitioning tool for landslide susceptibility assessment—a test in the Imera River basin (northern Sicily, Italy)

  2022cites this paper
• Development of Ebola virus disease prediction scores: Screening tools for Ebola suspects at the triage-point during an outbreak

  2022cites this paper
• Metabolomics study identified bile acids as potential biomarkers for gastric cancer: A case control study

  2022cites this paper
• Development and validation of a predictive scoring system for in-hospital mortality in COVID-19 Egyptian patients: a retrospective study

  2022cites this paper
• Predictive Score for Carbapenem-Resistant Gram-Negative Bacilli Sepsis: Single-Center Prospective Cohort Study

  2022influential citation
• Deep learning-based diagnosis of Alcohol use disorder (AUD) using EEG

  2022cites this paper
• Efficiency of consumer behaviour and digital ecosystem in the generation of the plastic waste toward the circular economy.

  2022cites this paper
• Clinical utility of automatic phenotype annotation in unstructured clinical notes: intensive care unit use

  2022cites this paper