The 2021 SIIM-FISABIO-RSNA Machine Learning COVID-19 Challenge: Annotation and Standard Exam Classification of COVID-19 Chest Radiographs

We describe the curation, annotation methodology, and characteristics of the dataset used in an artificial intelligence challenge for detection and localization of COVID-19 on chest radiographs. The chest radiographs were annotated by an international group of radiologists into four mutually exclusive categories, including “typical,” “indeterminate,” and “atypical appearance” for COVID-19, or “negative for pneumonia,” adapted from previously published guidelines, and bounding boxes were placed on airspace opacities. This dataset and respective annotations are available to researchers for academic and noncommercial use.

• Publication year

  2021

• Venue

  Journal of digital imaging

• Publication date

  2021-10-21

• Fields of study

  Medicine, Computer Science

• Identifiers
  DOI 10.1007/s10278-022-00706-8PMID 36171520PMCID 9518934
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar, PubMed

• No claims are published for this paper.

• No concepts are published for this paper.

• The RSNA International COVID-19 Open Radiology Database (RICORD)

  2021cited by this paper
• The RSNA Pulmonary Embolism CT Dataset.

  2021cited by this paper
• Multi-Radiologist User Study for Artificial Intelligence-Guided Grading of COVID-19 Lung Disease Severity on Chest Radiographs

  2021cited by this paper
• COVID-19 on Chest Radiographs: A Multireader Evaluation of an Artificial Intelligence System

  2020cited by this paper
• How Might AI and Chest Imaging Help Unravel COVID-19’s Mysteries?

  2020cited by this paper
• Construction of a Machine Learning Dataset through Collaboration: The RSNA 2019 Brain CT Hemorrhage Challenge.

  2020cited by this paper
• BIMCV COVID-19+: a large annotated dataset of RX and CT images from COVID-19 patients

  2020influential reference
• Review of Chest Radiograph Findings of COVID-19 Pneumonia and Suggested Reporting Language

  2020cited by this paper
• Automated Assessment and Tracking of COVID-19 Pulmonary Disease Severity on Chest Radiographs using Convolutional Siamese Neural Networks

  2020cited by this paper
• COVID-19 in CXR: From Detection and Severity Scoring to Patient Disease Monitoring

  2020cited by this paper
• Laboratory diagnosis of COVID-19

  2020cited by this paper
• Chest X-Ray Findings in 636 Ambulatory Patients with COVID-19 Presenting to an Urgent Care Center: A Normal Chest X-Ray Is no Guarantee

  2020cited by this paper
• AI for radiographic COVID-19 detection selects shortcuts over signal

  2020cited by this paper
• Initial chest radiographs and artificial intelligence (AI) predict clinical outcomes in COVID-19 patients: analysis of 697 Italian patients

  2020cited by this paper
• Performance of a Severity Score on Admission Chest Radiograph in Predicting Clinical Outcomes in Hospitalized Patients with Coronavirus Disease (COVID-19).

  2020cited by this paper
• Comparison of the characteristics, morbidity, and mortality of COVID-19 and seasonal influenza: a nationwide, population-based retrospective cohort study

  2020cited by this paper
• A Novel Coronavirus from Patients with Pneumonia in China, 2019

  2020cited by this paper
• The reproductive number of COVID-19 is higher compared to SARS coronavirus

  2020influential reference
• The Role of Chest Imaging in Patient Management During the COVID-19 Pandemic

  2020cited by this paper
• Radiographic severity index in COVID-19 pneumonia: relationship to age and sex in 783 Italian patients

  2020cited by this paper
• Frequency and Distribution of Chest Radiographic Findings in COVID-19 Positive Patients

  2019cited by this paper
• Challenges Related to Artificial Intelligence Research in Medical Imaging and the Importance of Image Analysis Competitions.

  2019cited by this paper
• Crowdsourcing pneumothorax annotations using machine learning annotations on the NIH chest X-ray dataset

  2019cited by this paper
• The RSNA Pediatric Bone Age Machine Learning Challenge.

  2019cited by this paper
• Automated detection of moderate and large pneumothorax on frontal chest X-rays using deep convolutional neural networks: A retrospective study

  2018cited by this paper
• Automatic Lung Cancer Prediction from Chest X-ray Images Using the Deep Learning Approach

  2018cited by this paper
• CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning

  2017cited by this paper
• Deep Learning at Chest Radiography: Automated Classification of Pulmonary Tuberculosis by Using Convolutional Neural Networks.

  2017cited by this paper
• The Berlin definition of ARDS: an expanded rationale, justification, and supplementary material

  2012cited by this paper
• Infectious Diseases Society of America/American Thoracic Society Consensus Guidelines on the Management of Community-Acquired Pneumonia in Adults

  2007cited by this paper
• Guidelines, Criteria, and Rules of Thumb for Evaluating Normed and Standardized Assessment Instruments in Psychology.

  1994influential reference
• Methods for Detecting Pulmonary Edema

  1985cited by this paper

• Expert-Guided Explainable Few-Shot Learning with Active Sample Selection for Medical Image Analysis

  2026cites this paper
• Computer Vision-Based Medical Imaging Techniques: Past, Present, and Future

  2026cites this paper
• Learning to Select Like Humans: Explainable Active Learning for Medical Imaging

  2026cites this paper
• Multi-Modal Explainable Medical AI Assistant for Trustworthy Human-AI Collaboration

  2025cites this paper
• DICA-Net: optimizing chest X-ray classification with attention U-Net and pigeon local search

  2025cites this paper
• Explained Deep Learning Framework for COVID-19 Detection in Volumetric CT Images Aligned with the British Society of Thoracic Imaging Reporting Guidance: A Pilot Study

  2025cites this paper
• Chest X-Ray Foundation Model With Global and Local Representations Integration

  2025cites this paper
• CXR-LLaVA: a multimodal large language model for interpreting chest X-ray images

  2025cites this paper
• Algorithm improvement based on YOLOv10 for pneumonia x-ray analysis

  2025cites this paper
• SPCF-YOLO: An Efficient Feature Optimization Model for Real-Time Lung Nodule Detection

  2025cites this paper
• Pitfalls and Best Practices in Evaluation of AI Algorithmic Biases in Radiology.

  2025cites this paper
• Multi-task supervised contrastive learning for chest X-ray diagnosis: A two-stage hierarchical classification framework for COVID-19 diagnosis

  2024cites this paper
• Tensor factor adjustment for image classification with pervasive noises

  2024cites this paper
• Self-eXplainable AI for Medical Image Analysis: A Survey and New Outlooks

  2024cites this paper
• Improving Fairness of Automated Chest Radiograph Diagnosis by Contrastive Learning.

  2024cites this paper
• Ensemble of Convolutional Neural Networks for COVID-19 Localization on Chest X-ray Images

  2024cites this paper
• Generalisation challenges in deep learning models for medical imagery: insights from external validation of COVID-19 classifiers

  2024cites this paper
• iWBFCxr: improved weighted box fusion for analyzing chest x ray images for enhancing thoracic abnormalities

  2024influential citation
• Deep Learning for Pneumonia Detection in Chest X-ray Images: A Comprehensive Survey

  2024cites this paper
• Augmentation strategies for an imbalanced learning problem on a novel COVID-19 severity dataset

  2023cites this paper
• Self-Supervised Curricular Deep Learning for Chest X-Ray Image Classification

  2023cites this paper
• Data drift in medical machine learning: implications and potential remedies.

  2023cites this paper
• Automatic Identification of Lung Opacities Due to COVID-19 from Chest X-ray Images—Focussing Attention on the Lungs

  2023influential citation
• Leveraging human expert image annotations to improve pneumonia differentiation through human knowledge distillation

  2023cites this paper
• Classification and Localization of Multi-Type Abnormalities on Chest X-Rays Images

  2023cites this paper
• Research on COVID-19 abnormalities on chest radiographs detection based on improved Yolov5

  2023cites this paper
• Improving model fairness in image-based computer-aided diagnosis

  2023cites this paper
• Analysis of generalizability on predicting COVID-19 from chest X-ray images using pre-trained deep models

  2023cites this paper
• A Robust Ensemble Convolutional Neural Networks for Diagnosing Chest Diseases

  2023influential citation
• BioEdge: Accelerating Object Detection in Bioimages with Edge-Based Distributed Inference

  2023cites this paper
• Pseudo-Labeling With Contrastive Perturbation Using CNN & ViT for Chest X-ray Classification

  2023cites this paper
• A Critical Assessment of Generative Models for Synthetic Data Augmentation on Limited Pneumonia X-ray Data

  2023cites this paper
• Challenges in COVID-19 Chest X-Ray Classification: Problematic Data or Ineffective Approaches?

  2022cites this paper
• Dark-field chest X-ray imaging for the assessment of COVID-19-pneumonia

  2022cites this paper
• Pneumonia and COVID-19 Detection in Chest X-rays Using Faster Region-Based Convolutional Neural Networks (Faster R-CNN)*

  2022cites this paper
• Machine Learning and Deep Learning in Cardiothoracic Imaging: A Scoping Review

  2022cites this paper
• Research and Development of Deep Learning Algorithms for the Classification of Pneumonia Type and Detection of Ground-Glass Loci on Radiological Images

  2022cites this paper
• GazeRadar: A Gaze and Radiomics-Guided Disease Localization Framework

  2022cites this paper
• Saliency Guided Adversarial Training for Learning Generalizable Features with Applications to Medical Imaging Classification System

  2022cites this paper
• Towards Trustworthy Healthcare AI: Attention-Based Feature Learning for COVID-19 Screening With Chest Radiography

  2022cites this paper
• Hybrid Learning System for Large-scale Medical Image Analysis

  2022cites this paper
• Identification and Localization of COVID-19 Abnormalities on Chest Radiographs using Ensembled Deep Neural Networks

  2022cites this paper
• Self-supervision and Multi-task Learning: Challenges in Fine-Grained COVID-19 Multi-class Classification from Chest X-rays

  2022cites this paper
• Detecting COVID-19 Opacity in X-ray Images Using YOLO and RetinaNet Ensemble

  2022cites this paper
• HINT: Hierarchical Neuron Concept Explainer

  2022cites this paper
• RadioTransformer: A Cascaded Global-Focal Transformer for Visual Attention-guided Disease Classification

  2022cites this paper
• A deep learning approach for COVID-19 screening and localization on chest x-ray images

  2022cites this paper
• Pneumonia and COVID-19 Classification in Chest X-rays Using Faster Region-Based Convolutional Neural Networks (Faster R-CNN)

  2022influential citation