A direct measure of discriminant and characteristic capability for classifier building and assessment

Performance measures are used in various stages of the process aimed at solving a classification problem. Unfortunately, most of these measures are in fact biased, meaning that they strictly depend on the class ratio - i.e. on the imbalance between negative and positive samples. After pointing to the source of bias for the best known measures, novel unbiased measures are defined which are able to capture the concepts of discriminant and characteristic capability. The combined use of these measures can give important information to researchers involved in machine learning or pattern recognition tasks, in particular for classifier performance assessment and feature selection.

• Publication year

  2015

• Venue

  Information Sciences

• Publication date

  2015-12-20

• Fields of study

  Computer Science

• Identifiers
  DOI 10.1016/j.ins.2015.07.028
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Conditional Likelihood Maximisation: A Unifying Framework for Information Theoretic Feature Selection

  2012cited by this paper
• A Unified View of Performance Metrics: Translating Threshold Choice into Expected Classification Loss C` Esar Ferri

  2012cited by this paper
• Detecting Novel Associations in Large Data Sets

  2011cited by this paper
• A Coherent Interpretation of AUC as a Measure of Aggregated Classification Performance

  2011cited by this paper
• A Survey on Graphical Methods for Classification Predictive Performance Evaluation

  2011cited by this paper
• Consistency measures for feature selection

  2008cited by this paper
• Measuring and testing dependence by correlation of distances

  2007cited by this paper
• Efiective Linear Discriminant Analysis for High Dimensional, Low Sample Size Data

  2007cited by this paper
• Discriminant analysis in correlation similarity measure space

  2007cited by this paper
• Searching for Interacting Features

  2007cited by this paper
• An introduction to ROC analysis

  2006cited by this paper
• Local Fisher discriminant analysis for supervised dimensionality reduction

  2006cited by this paper
• On Bias, Variance, 0/1—Loss, and the Curse-of-Dimensionality

  2004cited by this paper
• RCV1: A New Benchmark Collection for Text Categorization Research

  2004cited by this paper
• Canonical Correlation Analysis: An Overview with Application to Learning Methods

  2004cited by this paper
• The Geometry of ROC Space: Understanding Machine Learning Metrics through ROC Isometrics

  2003influential reference
• An Introduction to Variable and Feature Selection

  2003cited by this paper
• Consistency-based search in feature selection

  2003cited by this paper
• Statistical behavior and consistency of classification methods based on convex risk minimization

  2003cited by this paper
• Feature selection based on information theory, consistency and separability indices

  2002cited by this paper
• Blind separation of sources, Part I" An adaptive algorithm based on neuromimetic architecture

  2002cited by this paper
• Explicitly representing expected cost: an alternative to ROC representation

  2000cited by this paper
• Consistency Based Feature Selection

  2000cited by this paper
• A Unified Bias-Variance Decomposition for Zero-One and Squared Loss

  2000cited by this paper
• Blind Separation of Sources

  1999cited by this paper
• Construction and Assessment of Classification Rules

  1999cited by this paper
• Selection of Relevant Features and Examples in Machine Learning

  1997cited by this paper
• The use of the area under the ROC curve in the evaluation of machine learning algorithms

  1997cited by this paper
• Feature Selection and Classification - A Probabilistic Wrapper Approach

  1996cited by this paper
• Toward Optimal Feature Selection

  1996cited by this paper
• Efficient Algorithms for Minimizing Cross Validation Error

  1994cited by this paper
• Irrelevant Features and the Subset Selection Problem

  1994cited by this paper
• Using mutual information for selecting features in supervised neural net learning

  1994cited by this paper
• On bias.

  1994cited by this paper
• A critical investigation of recall and precision as measures of retrieval system performance

  1989cited by this paper
• 35 Use of distance measures, information measures and error bounds in feature evaluation

  1982cited by this paper
• Classification, pattern recognition, and reduction of dimensionality

  1982cited by this paper
• VERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITY

  1950cited by this paper
• Human behavior and the principle of least effort

  1949cited by this paper
• THE USE OF MULTIPLE MEASUREMENTS IN TAXONOMIC PROBLEMS

  1936cited by this paper
• VII. Note on regression and inheritance in the case of two parents

  year unknowncited by this paper

• Devising novel performance measures for assessing the behavior of multilayer perceptrons trained on regression tasks

  2023cites this paper
• Assessing Feature Importance for Short-Term Prediction of Electricity Demand in Medium-Voltage Loads

  2022cites this paper
• Using phidelta diagrams to discover relevant patterns in multilayer perceptrons

  2020cites this paper
• Using \varphi -\delta Diagrams on Web Data

  2018cites this paper
• A two-tiered 2D visual tool for assessing classifier performance

  2018cites this paper
• Phi-Delta-Diagrams: Software Implementation of a Visual Tool for Assessing Classifier and Feature Performance

  2018influential citation
• How Complex Is Your Classification Problem?

  2018cites this paper
• Medical decision support system for extremely imbalanced datasets

  2017cites this paper
• Automated Taxonomy Building by Adopting Discriminant and Characteristic Capabilities

  2016cites this paper
• Stopwords Identification by Means of Characteristic and Discriminant Analysis

  2015cites this paper
• Analysis of Term Roles Along Taxonomy Nodes by Adopting Discriminant and Characteristic Capabilities

  2015cites this paper