An interactive algorithm for asking and incorporating feature feedback into support vector machines

Standard machine learning techniques typically require ample training data in the form of labeled instances. In many situations it may be too tedious or costly to obtain sufficient labeled data for adequate classifier performance. However, in text classification, humans can easily guess the relevance of features, that is, words that are indicative of a topic, thereby enabling the classifier to focus its feature weights more appropriately in the absence of sufficient labeled data. We will describe an algorithm for tandem learning that begins with a couple of labeled instances, and then at each iteration recommends features and instances for a human to label. Tandem learning using an "oracle" results in much better performance than learning on only features or only instances. We find that humans can emulate the oracle to an extent that results in performance (accuracy) comparable to that of the oracle. Our unique experimental design helps factor out system error from human error, leading to a better understanding of when and why interactive feature selection works.

• Publication year

  2007

• Venue

  Annual International ACM SIGIR Conference on Research and Development in Information Retrieval

• Publication date

  2007-07-23

• Fields of study

  Computer Science

• Identifiers
  DOI 10.1145/1277741.1277758
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Topic Detection and Tracking

  2009cited by this paper
• Tandem learning: a learning framework for document categorization

  2007cited by this paper
• Constructing informative prior distributions from domain knowledge in text classification

  2006cited by this paper
• Active Learning with Feedback on Features and Instances

  2006cited by this paper
• Text clustering with extended user feedback

  2006cited by this paper
• Indri : A language-model based search engine for complex queries ( extended version )

  2005cited by this paper
• Incorporating prior knowledge with weighted margin support vector machines

  2004influential reference
• Document Classification Through Interactive Supervision of Document and Term Labels

  2004cited by this paper
• RCV1: A New Benchmark Collection for Text Categorization Research

  2004cited by this paper
• Re-examining the potential effectiveness of interactive query expansion

  2003cited by this paper
• Incorporating Prior Knowledge into Boosting

  2002cited by this paper
• Feature Selection Using Linear Support Vector Machines

  2002cited by this paper
• The Reuters Corpus Volume 1 -from Yesterday’s News to Tomorrow’s Language Resources

  2002cited by this paper
• Machine learning in automated text categorization

  2001cited by this paper
• On feature distributional clustering for text categorization

  2001cited by this paper
• Learning with Kernels: support vector machines, regularization, optimization, and beyond

  2001influential reference
• Active learning: theory and applications

  2001cited by this paper
• Support Vector Machine Active Learning with Applications to Text Classification

  2000cited by this paper
• Text Categorization with Support Vector Machines: Learning with Many Relevant Features

  1999cited by this paper
• The potential and actual effectiveness of interactive query expansion

  1997influential reference
• NewsWeeder: Learning to Filter Netnews

  1995influential reference
• Improving Generalization with Active Learning

  1994cited by this paper
• Heterogeneous Uncertainty Sampling for Supervised Learning

  1994influential reference
• Computational learning theory: survey and selected bibliography

  1992cited by this paper
• Experiments with query acquisition and use in document retrieval systems

  1989cited by this paper
• The measurement of observer agreement for categorical data.

  1977influential reference
• A Coefficient of Agreement for Nominal Scales

  1960cited by this paper
• and as an in

  year unknowncited by this paper

• Leveraging explanations in interactive machine learning: An overview

  2022cites this paper
• Machine Coaching with Proxy Coaches

  2022cites this paper
• LIMEADE: A General Framework for Explanation-Based Human Tuning of Opaque Machine Learners

  2021cites this paper
• Active Learning++: Incorporating Annotator's Rationale using Local Model Explanation

  2020cites this paper
• LIMEADE: From AI Explanations to Advice Taking

  2020cites this paper
• Local Decision Pitfalls in Interactive Machine Learning

  2019cites this paper
• Explanatory Interactive Machine Learning

  2019cites this paper
• A Unified Approach on Active Learning Dual Supervision

  2019cites this paper
• Bridging Text Visualization and Mining: A Task-Driven Survey

  2019cites this paper
• Dimension Reduction and Clustering for Interactive Visual Analytics

  2019cites this paper
• Interactive Machine Learning with Applications in Health Informatics

  2018cites this paper
• Learning with rationales for document classification

  2018cites this paper
• "Why Should I Trust Interactive Learners?" Explaining Interactive Queries of Classifiers to Users

  2018cites this paper
• Human Guided Linear Regression With Feature-Level Constraints

  2018cites this paper
• Active inference for dynamic Bayesian networks with an application to tissue engineering

  2017cites this paper
• Term selection in information retrieval

  2016cites this paper
• Active learning: an empirical study of common baselines

  2016cites this paper
• Towards Learning with Feature-Based Explanations for Document Classification

  2016cites this paper
• Document analysis by means of data mining techniques

  2015cites this paper
• Robustly Leveraging Prior Knowledge in Text Classification

  2015cites this paper
• Active Learning with Rationales for Text Classification

  2015cites this paper
• Comparing Arabic NLP tools for Hadith Classification

  2015cites this paper
• Learning Using 1-Local Membership Queries

  2015cites this paper
• Text Analytics of Social Media: Sentiment Analysis, Event Detection and Summarization

  2014cites this paper
• Improving Arabic Texts Morphological Disambiguation Using a Possibilistic Classifier

  2014cites this paper
• Learning from Labeled Features for Document Filtering

  2014cites this paper
• Legal keyword extraction and decision categorization : a case study on italian civil case law 1

  2014cites this paper
• WS2F: A weakly supervised framework for data stream filtering

  2014cites this paper
• Multi-label Classification of Croatian Legal Documents Using EuroVoc Thesaurus

  2014cites this paper
• A comparison of SVM and RVM for Document Classification

  2013cites this paper
• Incorporating Labeled Features into Image Classification using Generalized Expectation

  2013cites this paper
• Teaching with Examples in A Real Environment

  2013cites this paper
• End-user feature labeling: Supervised and semi-supervised approaches based on locally-weighted logistic regression

  2013influential citation
• A Simple Study of Webpage Text Classification Algorithms for Arabic and English Languages

  2013cites this paper
• Clustering Documents Along Multiple Dimensions

  2012cites this paper
• Regroup: interactive machine learning for on-demand group creation in social networks

  2012cites this paper
• End-user interactions with intelligent and autonomous systems

  2012cites this paper
• Cost-Sensitive Multi-strategy Active Annotation for Text Classification

  2012cites this paper
• A Survey of Text Classification Algorithms

  2012cites this paper
• Towards recognizing "cool": can end users help computer vision recognize subjective attributes of objects in images?

  2012cites this paper
• Manifold Adaptive Experimental Design for Text Categorization

  2012cites this paper
• Minimizing The Costs in Generalized Interactive Annotation Learning

  2012influential citation
• Mining Text Data

  2012cites this paper
• End-User Feature Labeling via Locally Weighted Logistic Regression

  2011influential citation
• Filtering semi-structured documents based on faceted feedback

  2011cites this paper
• End-user feature labeling: a locally-weighted regression approach

  2011influential citation
• A Non-negative Matrix Factorization Based Approach for Active Dual Supervision from Document and Word Labels

  2011cites this paper
• Designing for effective end-user interaction with machine learning

  2011cites this paper
• L1 vs. L2 Regularization in Text Classification when Learning from Labeled Features

  2011cites this paper
• Generalized expectation criteria for lightly supervised learning

  2011influential citation
• Towards subjectifying text clustering

  2010cites this paper
• Multi-faceted clustering: spectral and probabilistic approaches

  2010cites this paper
• Active learning for biomedical citation screening

  2010cites this paper
• A Unified Approach to Active Dual Supervision for Labeling Features and Examples

  2010cites this paper
• Guided Feature Labeling for Budget-Sensitive Learning Under Extreme Class Imbalance

  2010cites this paper
• Which Clustering Do You Want? Inducing Your Ideal Clustering with Minimal Feedback

  2010cites this paper
• A novel learning for image retrieval based on both keyword feature and instance feedback

  2010cites this paper
• End-user feature engineering in the presence of class imbalance

  2009cites this paper
• Topic-wise, Sentiment-wise, or Otherwise? Identifying the Hidden Dimension for Unsupervised Text Classification

  2009cites this paper
• Interactive Feature Space Construction using Semantic Information

  2009cites this paper
• Active Learning by Labeling Features

  2009influential citation
• Supervised Dual-PLSA for Personalized SMS Filtering

  2009cites this paper
• Beyond Feature Relevance: Incorporating Rich User Feedback Into Interactive Machine Learning Applications

  2009cites this paper
• Interactive Learning Protocols for Natural Language Applications

  2009cites this paper
• On the Empirical Complexity of Text Classification Problems

  2009cites this paper
• Multi-layer features based personalized spam filtering

  2009cites this paper
• Uncertainty sampling and transductive experimental design for active dual supervision

  2009influential citation
• Feature Construction Methods : A Survey

  2009cites this paper
• Active Dual Supervision: Reducing the Cost of Annotating Examples and Features

  2009cites this paper
• Modeling Annotators: A Generative Approach to Learning from Annotator Rationales

  2008cites this paper
• Learning from labeled features using generalized expectation criteria

  2008influential citation