Active Dual Supervision: Reducing the Cost of Annotating Examples and Features

When faced with the task of building machine learning or NLP models, it is often worthwhile to turn to active learning to obtain human annotations at minimal costs. Traditional active learning schemes query a human for labels of intelligently chosen examples. However, human effort can also be expended in collecting alternative forms of annotations. For example, one may attempt to learn a text classifier by labeling class-indicating words, instead of, or in addition to, documents. Learning from two different kinds of supervision brings a new, unexplored dimension to the problem of active learning. In this paper, we demonstrate the value of such active dual supervision in the context of sentiment analysis. We show how interleaving queries for both documents and words significantly reduces human effort -- more than what is possible through traditional one-dimensional active learning, or by passive combinations of supervisory inputs.

• Publication year

  2009

• Venue

  HLT-NAACL 2009

• Publication date

  2009-06-05

• Fields of study

  Computer Science

• Identifiers
  DOI 10.3115/1564131.1564142
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Budgeted Learning of Naive-Bayes Classifiers

  2012cited by this paper
• Uncertainty sampling and transductive experimental design for active dual supervision

  2009cited by this paper
• Sentiment analysis of blogs by combining lexical knowledge with text classification

  2009influential reference
• Active Feature-Value Acquisition

  2009cited by this paper
• Learning from labeled features using generalized expectation criteria

  2008cited by this paper
• Regularized Co-Clustering with Dual Supervision

  2008cited by this paper
• Modeling Annotators: A Generative Approach to Learning from Annotator Rationales

  2008cited by this paper
• Document-Word Co-regularization for Semi-supervised Sentiment Analysis

  2008cited by this paper
• Data acquisition and cost-effective predictive modeling: targeting offers for electronic commerce

  2007cited by this paper
• An interactive algorithm for asking and incorporating feature feedback into support vector machines

  2007cited by this paper
• Constructing informative prior distributions from domain knowledge in text classification

  2006cited by this paper
• Active Learning with Feedback on Features and Instances

  2006cited by this paper
• An expected utility approach to active feature-value acquisition

  2005cited by this paper
• Diverse ensembles for active learning

  2004cited by this paper
• Text Classification by Labeling Words

  2004cited by this paper
• Document Classification Through Interactive Supervision of Document and Term Labels

  2004cited by this paper
• Incorporating prior knowledge with weighted margin support vector machines

  2004cited by this paper
• Budgeted learning of nailve-bayes classifiers

  2002cited by this paper
• Thumbs up? Sentiment Classification using Machine Learning Techniques

  2002cited by this paper
• Incorporating Prior Knowledge into Boosting

  2002cited by this paper
• Toward Optimal Active Learning through Sampling Estimation of Error Reduction

  2001cited by this paper
• Support Vector Machine Active Learning with Applications to Text Classification

  2000cited by this paper
• ‘Combining Probability Distributions from Experts in Risk Analysis’

  2000cited by this paper
• A comparison of event models for naive bayes text classification

  1998cited by this paper
• Improving Generalization with Active Learning

  1994cited by this paper
• Heterogeneous Uncertainty Sampling for Supervised Learning

  1994cited by this paper

• Context-Aware Feature Selection and Classification

  2023cites this paper
• A Survey of Active Learning for Natural Language Processing

  2022cites this paper
• Hybrid Graph Neural Network Model Design and Modeling Reasoning for Text Feature Extraction and Recognition

  2022cites this paper
• Want To Reduce Labeling Cost? GPT-3 Can Help

  2021cites this paper
• Active neural learners for text with dual supervision

  2020cites this paper
• Interactive Multiagent Adaptation of Individual Classification Models for Decision Support

  2019cites this paper
• Interactive Machine Learning with Applications in Health Informatics

  2018cites this paper
• Complementing Machine Learning Classifiers via Dynamic Symbolic Execution: "Human vs. Bot Generated" Tweets

  2018cites this paper
• Learning with rationales for document classification

  2018influential citation
• Active inference for dynamic Bayesian networks with an application to tissue engineering

  2017cites this paper
• Prioritized active learning for malicious URL detection using weighted text-based features

  2017cites this paper
• Challenges of Reliable, Realistic and Comparable Active Learning Evaluation

  2017cites this paper
• Active Learning Strategy for Text Categorization Based on Support Vectors Relative Positioning

  2017cites this paper
• Active learning: an empirical study of common baselines

  2016cites this paper
• Towards Learning with Feature-Based Explanations for Document Classification

  2016cites this paper
• From Active Learning to Dedicated Collaborative Interactive Learning

  2016cites this paper
• A New Vision of Collaborative Active Learning

  2015cites this paper
• Active Learning with Rationales for Text Classification

  2015influential citation
• Personalized classifiers: evolving a classifier from a large reference knowledge graph

  2014cites this paper
• Class Imbalance and Active Learning

  2013cites this paper
• 6 IMBALANCE AND ACTIVE

  2013cites this paper
• CLASS IMBALANCE AND ACTIVE LEARNING

  2013cites this paper
• Active Learning on Sentiment Classification by Selecting Both Words and Documents

  2012cites this paper
• Minimizing The Costs in Generalized Interactive Annotation Learning

  2012cites this paper
• Integrating instance-level and attribute-level knowledge into document clustering

  2011cites this paper
• A transfer approach to detecting disease reporting events in blog social media

  2011cites this paper
• From Theories to Queries

  2011cites this paper
• A Non-negative Matrix Factorization Based Approach for Active Dual Supervision from Document and Word Labels

  2011cites this paper
• Class Imbalance and Active Learning

  2011cites this paper
• Guided Feature Labeling for Budget-Sensitive Learning Under Extreme Class Imbalance

  2010influential citation
• Chapter 1 Selective Data Acquisition for Machine Learning

  2010cites this paper
• A Unified Approach to Active Dual Supervision for Labeling Features and Examples

  2010cites this paper
• Uncertainty sampling and transductive experimental design for active dual supervision

  2009cites this paper