A shared task involving multi-label classification of clinical free text

This paper reports on a shared task involving the assignment of ICD-9-CM codes to radiology reports. Two features distinguished this task from previous shared tasks in the biomedical domain. One is that it resulted in the first freely distributable corpus of fully anonymized clinical text. This resource is permanently available and will (we hope) facilitate future research. The other key feature of the task is that it required categorization with respect to a large and commercially significant set of labels. The number of participants was larger than in any previous biomedical challenge task. We describe the data production process and the evaluation measures, and give a preliminary analysis of the results. Many systems performed at levels approaching the inter-coder agreement, suggesting that human-like performance on this task is within the reach of currently available technologies.

• Publication year

  2007

• Venue

  BioNLP@ACL

• Publication date

  2007-06-29

• Fields of study

  Medicine, Computer Science

• Identifiers
  DOI 10.3115/1572392.1572411
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Second i2b2 workshop on natural language processing challenges for clinical records.

  2008cited by this paper
• TREC 2007 Genomics Track Overview

  2007cited by this paper
• Role of Local Context in Automatic Deidentification of Ungrammatical, Fragmented Text

  2006cited by this paper
• Text Mining for Biology and Biomedicine

  2006cited by this paper
• Preparing Clinical Text for Use in Biomedical Research

  2006cited by this paper
• Metric and Euclidean Properties of Diss imilari ty Coefficients

  2005cited by this paper
• Sample size and power estimation for studies with health related quality of life outcomes: a comparison of four methods using the SF-36

  2004cited by this paper
• Development of a Pediatric Text-Corpus for Part-of-Speech Tagging

  2004cited by this paper
• Natural language processing for online applications; text retrieval, extraction and categorization

  2004cited by this paper
• Multi-label Semantic Scene Classfication

  2003cited by this paper
• Text Boundary Detection of Medical Reports

  2002cited by this paper
• Two biomedical sublanguages: a description based on the theories of Zellig Harris

  2002cited by this paper
• The sublanguage of cross-coverage

  2002cited by this paper
• Crossing the Quality Chasm: A New Health System for the 21st Century

  2002cited by this paper
• Natural Language Processing for Online Applications: Text Retrieval, Extraction & Categorization

  2002cited by this paper
• Parsing techniques

  1997cited by this paper
• Metric and Euclidean properties of dissimilarity coefficients

  1986cited by this paper
• Sublanguage : studies of language in restricted semantic domains

  1982cited by this paper
• Chapter 2. Automatic Information Formatting of a Medical Sublanguage

  1982cited by this paper

• On Evaluation of Unsupervised Feature Selection for Pattern Classification

  2026cites this paper
• Fuzzy bifocal disambiguation for partial multi-label learning

  2025cites this paper
• A neuro-inspired approach for Continual Multi-Label Learning with evolving spiking networks

  2025cites this paper
• Partial multi-label learning with label and classifier correlations

  2025cites this paper
• From Sigmoid to SoftProb: A novel output activation function for multi-label learning

  2025cites this paper
• A Unified Feature Selection Framework Combining Mutual Information and Regression Optimization for Multi-Label Learning

  2025cites this paper
• Deep label relevance and label ambiguity based multi-label feature selection for text classification

  2025cites this paper
• CCN+: A neuro-symbolic framework for deep learning with requirements

  2024cites this paper
• Towards Robust Multi-Label Learning against Dirty Label Noise

  2024cites this paper
• A comprehensive review of existing corpora and methods for creating annotated corpora for event extraction tasks

  2024cites this paper
• Memetic multilabel feature selection using pruned refinement process

  2024cites this paper
• Czech medical coding assistant based on transformer networks

  2024cites this paper
• AI-Powered Medical Record Summarization for Diagnosis and Physician Training

  2024cites this paper
• Aligning AI Research with the Needs of Clinical Coding Workflows: Eight Recommendations Based on US Data Analysis and Critical Review

  2024cites this paper
• Performance evaluation of seven multi-label classification methods on real-world patent and publication datasets

  2024cites this paper
• Enhancing Multi-Label Classification Through Deep Extra-Trees and Transformation Techniques

  2024cites this paper
• Label dependency modeling in Multi-Label Naïve Bayes through input space expansion

  2024cites this paper
• Application of Label Correlation in Multi-Label Classification: A Survey

  2024cites this paper
• Automated Medical Record Summarization for Diagnosis and Training Doctors Using AI

  2024cites this paper
• Partial label feature selection based on noisy manifold and label distribution

  2024cites this paper
• Automating the overburdened clinical coding system: challenges and next steps

  2023cites this paper
• Rare Codes Count: Mining Inter-code Relations for Long-tail Clinical Text Classification

  2023cites this paper
• Goal Driven Discovery of Distributional Differences via Language Descriptions

  2023cites this paper
• mFILS: Tri-Selection via Convex and Nonconvex Regularizations

  2023cites this paper
• Ranking with Slot Constraints

  2023cites this paper
• Improving Automatic KCD Coding: Introducing the KoDAK and an Optimized Tokenization Method for Korean Clinical Documents

  2023cites this paper
• Development and validation of deep learning and BERT models for classification of lung cancer radiology reports

  2023cites this paper
• Explainable clinical coding with in-domain adapted transformers

  2023cites this paper
• Distributed Online Multi-Label Learning with Privacy Protection in Internet of Things

  2023cites this paper
• Using a Large Open Clinical Corpus for Improved ICD-10 Diagnosis Coding.

  2023cites this paper
• Knowledge Amalgamation for Multi-Label Classification via Label Dependency Transfer

  2023cites this paper
• Advancing Cancer Document Classification with R andom Forest

  2023cites this paper
• A correlation-based feature weighting filter for multi-label Naive Bayes

  2023cites this paper
• Multi-label feature selection based on label correlations and feature redundancy

  2022cites this paper
• Multi-label feature selection using geometric series of relevance matrix

  2022cites this paper
• AI-based ICD coding and classification approaches using discharge summaries: A systematic literature review

  2022cites this paper
• A Survey of Automated ICD Coding: Development, Challenges, and Applications

  2022cites this paper
• The role of machine learning in developing non-magnetic resonance imaging based biomarkers for multiple sclerosis: a systematic review

  2022cites this paper
• Code Synonyms Do Matter: Multiple Synonyms Matching Network for Automatic ICD Coding

  2022cites this paper
• Can Natural Language Processing and Artificial Intelligence Automate The Generation of Billing Codes From Operative Note Dictations?

  2022cites this paper
• A Unified Review of Deep Learning for Automated Medical Coding

  2022cites this paper
• Disease prediction based on multi-type data fusion from Chinese electronic health record.

  2022cites this paper
• Revisiting Transformer-based Models for Long Document Classification

  2022cites this paper
• OSLAT: Open Set Label Attention Transformer for Medical Entity Retrieval and Span Extraction

  2022cites this paper
• OSLAT: Open Set Label Attention Transformer for Medical Entity Span Extraction

  2022cites this paper
• Concise and interpretable multi-label rule sets

  2022cites this paper
• A fast unsupervised assignment of ICD codes with clinical notes through explanations

  2022cites this paper
• EvoSplit: An evolutionary approach to split a multi-label data set into disjoint subsets

  2021cites this paper
• Clinical Natural Language Processing for Radiation Oncology: A Review and Practical Primer.

  2021cites this paper
• ReliefE: feature ranking in high-dimensional spaces via manifold embeddings

  2021cites this paper
• Multi-Label Classification Neural Networks with Hard Logical Constraints

  2021cites this paper
• Saliency-Based Multilabel Linear Discriminant Analysis

  2021cites this paper
• Multi-label Diagnosis Classification of Swedish Discharge Summaries – ICD-10 Code Assignment Using KB-BERT

  2021cites this paper
• Fusion: Towards Automated ICD Coding via Feature Compression

  2021cites this paper
• Transformers for Clinical Coding in Spanish

  2021cites this paper
• The Pipeline for Standardizing Russian Unstructured Allergy Anamnesis Using FHIR AllergyIntolerance Resource

  2021cites this paper
• A Systematic Literature Review of Automated ICD Coding and Classification Systems using Discharge Summaries

  2021cites this paper
• Detection of Privacy Disclosure in the Medical Domain: A Survey

  2020cites this paper
• An evolutionary decomposition-based multi-objective feature selection for multi-label classification

  2020cites this paper
• Linguistic Typology Features from Text: Inferring the Sparse Features of World Atlas of Language Structures

  2020cites this paper
• Regularized Graph Convolutional Networks for Short Text Classification

  2020cites this paper
• Saliency-based Weighted Multi-label Linear Discriminant Analysis

  2020cites this paper
• Online Metric Learning for Multi-Label Classification

  2020cites this paper
• Two-level label recovery-based label embedding for multi-label classification with missing labels

  2020cites this paper
• Assistant diagnosis with Chinese electronic medical records based on CNN and BiLSTM with phrase-level and word-level attentions

  2020cites this paper
• Partial multi-label learning with noisy side information

  2020cites this paper
• Automatic Annotation of Narrative Radiology Reports

  2020cites this paper
• Classification of Syncope Cases in Norwegian Medical Records

  2020cites this paper
• An innovative multi-label learning based algorithm for city data computing

  2020cites this paper
• CREDO: Efficient and privacy-preserving multi-level medical pre-diagnosis based on ML-kNN

  2020cites this paper
• ICB-UMA at CLEF e-Health 2020 Task 1: Automatic ICD-10 coding in Spanish with BERT

  2020cites this paper
• Hybrid Noise-Oriented Multilabel Learning

  2020cites this paper
• Explainable Automated Coding of Clinical Notes using Hierarchical Label-wise Attention Networks and Label Embedding Initialisation

  2020cites this paper
• IXA-AAA at CLEF eHealth 2020 CodiEsp. Automatic Classification of Medical Records with Multi-label Classifiers and Similarity Match Coders

  2020cites this paper
• Artificial Intelligence Applications for Workflow, Process Optimization and Predictive Analytics.

  2020cites this paper
• NLNDE at CANTEMIST: Neural Sequence Labeling and Parsing Approaches for Clinical Concept Extraction

  2020cites this paper
• ICB-UMA at CANTEMIST 2020: Automatic ICD-O Coding in Spanish with BERT

  2020cites this paper
• Non-parametric predictive inference for solving multi-label classification

  2020cites this paper
• Multi-label feature ranking with ensemble methods

  2020cites this paper
• Assisting nurses in care documentation: from automated sentence classification to coherent document structures with subject headings

  2020cites this paper
• A Survey of Deep Active Learning

  2020cites this paper
• Feature ranking for semi-supervised learning

  2020cites this paper
• A Survey of Active Learning for Text Classification using Deep Neural Networks

  2020cites this paper
• Threshold Moving Approaches for Addressing the Class Imbalance Problem and their Application to Multi-label Classification

  2020cites this paper
• Privacy Guarantees for De-identifying Text Transformations

  2020cites this paper
• Multi-Population Genetic Algorithm for Multilabel Feature Selection Based on Label Complementary Communication

  2020cites this paper
• An empirical analysis of binary transformation strategies and base algorithms for multi-label learning

  2020cites this paper
• Supervised Learning for the ICD-10 Coding of French Clinical Narratives

  2020cites this paper
• Collaboration Based Multi-Label Propagation for Fraud Detection

  2020cites this paper
• Partial Multi-Label Learning via Multi-Subspace Representation

  2020cites this paper
• Capturing Joint Label Distribution for Multi-Label Classification Through Adversarial Learning

  2020cites this paper
• Overview of Automatic Clinical Coding: Annotations, Guidelines, and Solutions for non-English Clinical Cases at CodiEsp Track of CLEF eHealth 2020

  2020cites this paper
• Anonymization Framework for Securing Protected Health Information in a Complex Dataset of Medical Narratives

  2020cites this paper
• Automated ICD-10 Classification of Mortality Causes from Textual Records

  2020cites this paper
• Bayes metaclassifier and Soft-confusion-matrix classifier in the task of multi-label classification

  2019cites this paper
• An approximated decision-theoretic algorithm for minimization of the Tversky loss under the multi-label framework

  2019cites this paper
• Distributed multi-label learning on Apache Spark

  2019cites this paper
• Neural transfer learning for assigning diagnosis codes to EMRs

  2019cites this paper
• Interpretable deep learning to map diagnostic texts to ICD-10 codes

  2019cites this paper
• Short Text Classification using Graph Convolutional Network

  2019cites this paper