Multi-Criteria-based Active Learning for Named Entity Recognition

In this paper, we propose a multi-criteria-based active learning approach and effectively apply it to named entity recognition. Active learning targets to minimize the human annotation efforts by selecting examples for labeling. To maximize the contribution of the selected examples, we consider the multiple criteria: informativeness, representativeness and diversity and propose measures to quantify them. More comprehensively, we incorporate all the criteria using two selection strategies, both of which result in less labeling cost than single-criterion-based method. The results of the named entity recognition in both MUC-6 and GENIA show that the labeling cost can be reduced by at least 80% without degrading the performance.

• Publication year

  2004

• Venue

  Annual Meeting of the Association for Computational Linguistics

• Publication date

  2004-07-21

• Fields of study

  Computer Science

• Identifiers
  DOI 10.3115/1218955.1219030
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Statistical Learning Theory

  2021cited by this paper
• A Collaborative Ability Measurement for Co-training

  2004cited by this paper
• Recognizing names in biomedical texts: a machine learning approach

  2004cited by this paper
• Enhancing HMM-based biomedical named entity recognition by studying special phenomena

  2004cited by this paper
• Effective Adaptation of Hidden Markov Model-based Named Entity Recognizer for Biomedical Domain

  2003cited by this paper
• Incorporating Diversity in Active Learning with Support Vector Machines

  2003cited by this paper
• Two-Phase Biomedical NE Recognition based on SVMs

  2003cited by this paper
• Example Selection for Bootstrapping Statistical Parsers

  2003cited by this paper
• Active Learning Selection Strategies for Information Extraction

  2003cited by this paper
• Use of Support Vector Machines in Extended Named Entity Recognition

  2002cited by this paper
• The GENIA corpus: an annotated research abstract corpus in molecular biology domain

  2002cited by this paper
• Active Learning for Statistical Natural Language Parsing

  2002influential reference
• Named Entity Recognition using an HMM-based Chunk Tagger

  2002cited by this paper
• Tuning support vector machines for biomedical named entity recognition

  2002cited by this paper
• Learning to classify text using support vector machines - methods, theory and algorithms

  2002cited by this paper
• The Nature of Statistical Learning Theory

  2000influential reference
• Less is More: Active Learning with Support Vector Machines

  2000influential reference
• Rule Writing or Annotation: Cost-efficient Resource Usage for Base Noun Phrase Chunking

  2000cited by this paper
• Support Vector Machine Active Learning with Applications to Text Classification

  2000influential reference
• Sample Selection for Statistical Grammar Induction

  2000cited by this paper
• Active Learning for Natural Language Parsing and Information Extraction

  1999influential reference
• Committee-Based Sample Selection for Probabilistic Classifiers

  1999cited by this paper
• Making large scale SVM learning practical

  1998cited by this paper
• Employing Em in Pool-based Active Learning for Text Classiication

  1998cited by this paper
• Employing EM and Pool-Based Active Learning for Text Classification

  1998influential reference
• Selective Sampling Using the Query by Committee Algorithm

  1997cited by this paper
• Statistical methods for speech recognition

  1997cited by this paper
• Committee-Based Sampling For Training Probabilistic Classi(cid:12)ers

  1995cited by this paper
• Heterogeneous Uncertainty Sampling for Supervised Learning

  1994cited by this paper
• A sequential algorithm for training text classifiers

  1994cited by this paper
• Query by committee

  1992cited by this paper
• Considerations in dynamic time warping algorithms for discrete word recognition

  1978cited by this paper
• Minimum prediction residual principle applied to speech recognition

  1975cited by this paper
• A Dynamic Programming Approach to Continuous Speech Recognition

  1971cited by this paper

• OWNER — Toward Unsupervised Open-World Named Entity Recognition

  2025cites this paper
• Contrastive Preference-Guided Active Learning Approach Based on Ranking Correlation for Real-Time Safety Assessment

  2025cites this paper
• Self-driving laboratories with artificial intelligence: An overview of process systems engineering perspective

  2025cites this paper
• Audio-Enhanced Vision-Language Modeling with Latent Space Broadening for High Quality Data Expansion

  2025cites this paper
• ALPET: Active Few-shot Learning for Citation Worthiness Detection in Low-Resource Wikipedia Languages

  2025cites this paper
• Accelerated Deep Active Learning with Graph-based Sub- Sampling

  2024cites this paper
• SciCN: A Scientific Dataset for Chinese Named Entity Recognition

  2024cites this paper
• Multi-strategy active learning for power quality disturbance identification

  2024cites this paper
• Identifying Data Breaches in Dark Web through Prompt Active Learning

  2024cites this paper
• Named entity recognition of pharmacokinetic parameters in the scientific literature

  2024cites this paper
• Modeling of learning curves with applications to POS tagging

  2024cites this paper
• Combining Active Learning and Learning to Reject for Anomaly Detection

  2024cites this paper
• Few-shot Named Entity Recognition via Superposition Concept Discrimination

  2024cites this paper
• Automatic Few-Shot Selection on In-Context Learning for Aspect Term Extraction

  2024cites this paper
• Near-Optimal Active Learning for Multilingual Grapheme-to-Phoneme Conversion

  2023cites this paper
• Perturbation-based Active Learning for Question Answering

  2023cites this paper
• Semi-Automated Construction of Food Composition Knowledge Base

  2023cites this paper
• Divide & Conquer for Entailment-aware Multi-hop Evidence Retrieval

  2023cites this paper
• Learning Geolocation by Accurately Matching Customer Addresses via Graph based Active Learning

  2023cites this paper
• Active Learning for Video Classification with Frame Level Queries

  2023cites this paper
• Étude comparative des plongements lexicaux pour l’extraction d’entités nommées en français

  2023cites this paper
• Active Learning and Bayesian Optimization: A Unified Perspective to Learn with a Goal

  2023cites this paper
• MIWA: Mixed-Initiative Web Automation for Better User Control and Confidence

  2023cites this paper
• An Empirical Study on Active Learning for Multi-label Text Classification

  2023cites this paper
• The use of Active Learning systems for stimulus selection and response modelling in perception experiments

  2023cites this paper
• Active learning based on similarity level histogram and adaptive-scale sampling for very high resolution image classification

  2023cites this paper
• More Samples or More Prompts? Exploring Effective Few-Shot In-Context Learning for LLMs with In-Context Sampling

  2023cites this paper
• Interactive Multi-fidelity Learning for Cost-effective Adaptation of Language Model with Sparse Human Supervision

  2023cites this paper
• Data-efficient Active Learning for Structured Prediction with Partial Annotation and Self-Training

  2023cites this paper
• Stratégies d’apprentissage actif pour la reconnaissance d’entités nommées en français

  2023cites this paper
• A SHAP-based Active Learning Approach for Creating High-Quality Training Data

  2022cites this paper
• Automatic Document Selection for Efficient Encoder Pretraining

  2022cites this paper
• Surfing the modeling of PoS taggers in low-resource scenarios

  2022cites this paper
• Medical Named Entity Recognition Using Weakly Supervised Learning

  2022cites this paper
• A Survey of Active Learning for Natural Language Processing

  2022influential citation
• Asymptotic optimality for active learning processes

  2022cites this paper
• Looking Back on the Past: Active Learning With Historical Evaluation Results

  2022influential citation
• Pareto Optimization for Active Learning under Out-of-Distribution Data Scenarios

  2022influential citation
• From Limited Annotated Raw Material Data to Quality Production Data: A Case Study in the Milk Industry

  2021cites this paper
• Active Learning by Acquiring Contrastive Examples

  2021cites this paper
• ALLWAS: Active Learning on Language models in WASserstein space

  2021cites this paper
• Focusing on Possible Named Entities in Active Named Entity Label Acquisition

  2021cites this paper
• Active Learning for Argument Mining: A Practical Approach

  2021cites this paper
• The Impact of Active Learning Algorithm on a Cross-lingual model in a Persian Sentiment Task

  2021cites this paper
• The Named Entity Recognition of Chinese Cybersecurity Using an Active Learning Strategy

  2021cites this paper
• Reliable Estimates of Interpretable Cue Effects with Active Learning in Psycholinguistic Research

  2021cites this paper
• Multiple-criteria Based Active Learning with Fixed-size Determinantal Point Processes

  2021influential citation
• Leveraging Sequential Pattern Information for Active Learning from Sequential Data

  2021cites this paper
• A Goal Programming-Based Methodology for Machine Learning Model Selection Decisions: A Predictive Maintenance Application

  2021cites this paper
• Focusing on potential named entities during active label acquisition

  2021influential citation
• Using Pre-trained Language Model to Enhance Active Learning for Sentence Matching

  2021cites this paper
• A Comparative Survey: Benchmarking for Pool-based Active Learning

  2021cites this paper
• An Interactive Visualization Tool for Understanding Active Learning

  2021cites this paper
• A Survey of Active Learning for Text Classification using Deep Neural Networks

  2020cites this paper
• ALdataset: a benchmark for pool-based active learning

  2020cites this paper
• Active Learning for Air Quality Station Deployment

  2020cites this paper
• Cold-start Active Learning through Self-Supervised Language Modeling

  2020cites this paper
• Information Preparation with the Human in the Loop

  2020cites this paper
• Batch Mode Active Learning on the Riemannian Manifold for Automated Scoring of Nuclear Pleomorphism in Breast Cancer

  2020cites this paper
• Asking the Right Questions to the Right Users: Active Learning with Imperfect Oracles

  2020cites this paper
• Annotating Social Determinants of Health Using Active Learning, and Characterizing Determinants Using Neural Event Extraction

  2020cites this paper
• SAR Target Classification with Limited Data via Data Driven Active Learning

  2020cites this paper
• Finding the Sweet Spot: Batch Selection for One-Class Active Learning

  2020influential citation
• SinNer@Clef-Hipe2020 : Sinful adaptation of SotA models for Named Entity Recognition in French and German

  2020cites this paper
• Efficiently labelling sequences using semi-supervised active learning

  2020cites this paper
• Deep Active Learning for Anchor User Prediction

  2019cites this paper
• Active Generative Adversarial Network for Image Classification

  2019cites this paper
• Terminologies augmented recurrent neural network model for clinical named entity recognition

  2019cites this paper
• Named Entity Recognition With Deep Learning

  2019cites this paper
• Visualising and evaluating the effects of combining active learning with word embedding features

  2019cites this paper
• Merging Weak and Active Supervision for Semantic Parsing

  2019influential citation
• Tracing with Less Data: Active Learning for Classification-Based Traceability Link Recovery

  2019cites this paper
• Coner : A Collaborative Approach for Long-Tail Named Entity Recognition in Scienti c

  2019cites this paper
• A Generic Active Learning Framework for Class Imbalance Applications

  2019cites this paper
• Active Learning with n-ary Queries for Image Recognition

  2019cites this paper
• Practical, Efficient, and Customizable Active Learning for Named Entity Recognition in the Digital Humanities

  2019influential citation
• Coner: A Collaborative Approach for Long-Tail Named Entity Recognition in Scientific Publications

  2019cites this paper
• A Text Mining Pipeline Using Active and Deep Learning Aimed at Curating Information in Computational Neuroscience

  2019cites this paper
• A Method of Active Learning Based on Fuzzy Set Theory

  2019cites this paper
• Named Entity Recognition with Extremely Limited Data

  2018cites this paper
• Pool-Based Sequential Active Learning for Regression

  2018cites this paper
• Towards Practical Active Learning for Classification

  2018cites this paper
• A Text Mining Pipeline Using Active and Deep Learning Aimed at Curating Information in Computational Neuroscience

  2018cites this paper
• One Sense Per Document: Improve Name Finding in the Wild with Document-wide Context

  2018cites this paper
• Mixed-Initiative Active Learning for Generating Linguistic Insights in Question Classification

  2018cites this paper
• A novel active learning framework for classification: using weighted rank aggregation to achieve multiple query criteria

  2018influential citation
• Low-Cost Implementation of a Named Entity Recognition System for Voice-Activated Human-Appliance Interfaces in a Smart Home

  2018cites this paper
• Minimization of annotation work: diagnosis of mammographic masses via active learning

  2018influential citation
• Active Learning for New Domains in Natural Language Understanding

  2018cites this paper
• Learning More, with Less

  2017cites this paper
• Deep active learning for image classification

  2017cites this paper
• Clinical information extraction using small data: An active learning approach based on sequence representations and word embeddings

  2017cites this paper
• Multi-criteria decision making based architecture selection for single-hidden layer feedforward neural networks

  2017cites this paper
• Learning how to Active Learn: A Deep Reinforcement Learning Approach

  2017cites this paper
• Deep Active Learning for Named Entity Recognition

  2017cites this paper
• Sanremo’s Winner Is... Category-driven Selection Strategies for Active Learning

  2017cites this paper
• A Semi-automatic and low-cost method to learn patterns for named entity recognition*

  2017cites this paper
• Semantic Interpretation of Events in Live Soccer Commentaries

  2016cites this paper
• Machine Learning Approach for Multi-Layered Detection of Chemical Named Entities in Text

  2016cites this paper
• Named-Entity-Recognition (NER) for Tamil Language Using Margin-Infused Relaxed Algorithm (MIRA)

  2016cites this paper