Optimizing to Arbitrary NLP Metrics using Ensemble Selection

While there have been many successful applications of machine learning methods to tasks in NLP, learning algorithms are not typically designed to optimize NLP performance metrics. This paper evaluates an ensemble selection framework designed to optimize arbitrary metrics and automate the process of algorithm selection and parameter tuning. We report the results of experiments that instantiate the framework for three NLP tasks, using six learning algorithms, a wide variety of parameterizations, and 15 performance metrics. Based on our results, we make recommendations for subsequent machine-learning-based research for natural language learning.

• Publication year

  2005

• Venue

  Human Language Technology - The Baltic Perspectiv

• Publication date

  2005-10-06

• Fields of study

  Linguistics, Computer Science

• Identifiers
  DOI 10.3115/1220575.1220643
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Optimization issues in machine learning of coreference resolution

  2005cited by this paper
• Ensemble selection from libraries of models

  2004influential reference
• Improving Machine Learning Approaches to Noun Phrase Coreference Resolution

  2004influential reference
• Playing the Telephone Game: Determining the Hierarchical Structure of Perspective and Speech Expressions

  2004influential reference
• Recognizing and Organizing Opinions Expressed in the World Press

  2003cited by this paper
• Parameter optimization for machine-learning of word sense disambiguation

  2002cited by this paper
• Evaluation of Machine Learning Methods for Natural Language Processing Tasks

  2002cited by this paper
• Improving Machine Learning Approaches to Coreference Resolution

  2002influential reference
• A Machine Learning Approach to Coreference Resolution of Noun Phrases

  2001cited by this paper
• Probabilistic Outputs for Support vector Machines and Comparisons to Regularized Likelihood Methods

  1999cited by this paper
• Advances in kernel methods: support vector learning

  1999cited by this paper
• Making large scale SVM learning practical

  1998cited by this paper
• Algorithms for Scoring Coreference Chains

  1998cited by this paper
• TiMBL: Tilburg Memory-Based Learner, version 2.0, Reference guide

  1998cited by this paper
• Addressing the Curse of Imbalanced Training Sets: One-Sided Selection

  1997cited by this paper
• Experiments with a New Boosting Algorithm

  1996cited by this paper
• Bagging Predictors

  1996cited by this paper
• A Model-Theoretic Coreference Scoring Scheme

  1995cited by this paper
• Fast Effective Rule Induction

  1995cited by this paper
• Learning classification trees

  1992cited by this paper
• Algorithms for Scoring Coreference

  year unknowncited by this paper

• Analysis of sentiment in tweets addressed to a single domain-specific Twitter account: Comparison of model performance and explainability of predictions

  2021cites this paper
• Sentiment Analysis Main Tasks and Applications: A Survey

  2019cites this paper
• Collaborative Partitioning for Coreference Resolution

  2017cites this paper
• Deep Sequential and Structural Neural Models of Compositionality

  2017cites this paper
• QPLSA: Utilizing quad-tuples for aspect identification and rating

  2015cites this paper
• Opinion Mining with Deep Recurrent Neural Networks

  2014cites this paper
• Diagnóstico de glaucoma baseado em classificadores de aprendizagem de máquina utilizando dados do Spectral Domain-OCT e perimetria automatizada acromática

  2013cites this paper
• Domain Adaptation for Coreference Resolution: An Adaptive Ensemble Approach

  2012influential citation
• Extracting Opinion Expressions with semi-Markov Conditional Random Fields

  2012cites this paper
• Multi-metric optimization for coreference: The UniTN / IITP / Essex submission to the 2011 CONLL Shared Task

  2011cites this paper
• Single and multi-objective optimization for feature selection in anaphora resolution

  2011cites this paper
• Multiobjective Simulated Annealing Based Approach for Feature Selection in Anaphora Resolution

  2011cites this paper
• Empirical Methods for Fine-grained Opinion Extraction from Text

  2008influential citation
• Classifier Loss Under Metric Uncertainty

  2007cites this paper
• Identifying Expressions of Opinion in Context

  2007cites this paper
• Getting the Most Out of Ensemble Selection

  2006cites this paper
• Get out the vote: Determining support or opposition from Congressional floor-debate transcripts

  2006cites this paper
• Joint Extraction of Entities and Relations for Opinion Recognition

  2006cites this paper
• Automatic Analysis of Document Sentiment

  2006cites this paper
• Ensemble Selection from . . .

  2004cites this paper