On Fairness and Calibration

The machine learning community has become increasingly concerned with the potential for bias and discrimination in predictive models. This has motivated a growing line of work on what it means for a classification procedure to be "fair." In this paper, we investigate the tension between minimizing error disparity across different population groups while maintaining calibrated probability estimates. We show that calibration is compatible only with a single error constraint (i.e. equal false-negatives rates across groups), and show that any algorithm that satisfies this relaxation is no better than randomizing a percentage of predictions for an existing classifier. These unsettling findings, which extend and generalize existing results, are empirically confirmed on several datasets.

• Publication year

  2017

• Venue

  Neural Information Processing Systems

• Publication date

  2017-09-06

• Fields of study

  Mathematics, Computer Science

• Identifiers
  DOI 10.1145/3603195.3603198arXiv 1709.02012
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Learning Non-Discriminatory Predictors

  2017cited by this paper
• Meritocratic Fairness for Cross-Population Selection

  2017cited by this paper
• An algorithm for removing sensitive information: Application to race-independent recidivism prediction

  2017cited by this paper
• Algorithmic Decision Making and the Cost of Fairness

  2017cited by this paper
• Avoiding Discrimination through Causal Reasoning

  2017cited by this paper
• On Calibration of Modern Neural Networks

  2017cited by this paper
• Fairness in Criminal Justice Risk Assessments: The State of the Art

  2017influential reference
• Counterfactual Fairness

  2017cited by this paper
• A Primer on Fairness in Criminal Justice Risk Assessments

  2016cited by this paper
• Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction Instruments

  2016influential reference
• Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings

  2016cited by this paper
• Fairness Beyond Disparate Treatment & Disparate Impact: Learning Classification without Disparate Mistreatment

  2016cited by this paper
• Satisfying Real-world Goals with Dataset Constraints

  2016cited by this paper
• Equality of Opportunity in Supervised Learning

  2016influential reference
• Fairness in Learning: Classic and Contextual Bandits

  2016cited by this paper
• Inherent Trade-Offs in the Fair Determination of Risk Scores

  2016influential reference
• False Positives, False Negatives, and False Analyses: A Rejoinder to "Machine Bias: There's Software Used across the Country to Predict Future Criminals. and It's Biased against Blacks"

  2016influential reference
• Fair Algorithms for Infinite and Contextual Bandits

  2016cited by this paper
• Assessing calibration of prognostic risk scores

  2016cited by this paper
• COMPAS Risk Scales : Demonstrating Accuracy Equity and Predictive Parity Performance of the COMPAS Risk Scales in Broward County

  2016influential reference
• Big Data's Disparate Impact

  2016cited by this paper
• Censoring Representations with an Adversary

  2015cited by this paper
• On the relation between accuracy and fairness in binary classification

  2015cited by this paper
• Learning Fair Classifiers

  2015cited by this paper
• Fairness Constraints: Mechanisms for Fair Classification

  2015cited by this paper
• The Variational Fair Autoencoder

  2015cited by this paper
• Certifying and Removing Disparate Impact

  2014cited by this paper
• A multidisciplinary survey on discrimination analysis

  2013cited by this paper
• Learning Fair Representations

  2013cited by this paper
• Fairness-aware Learning through Regularization Approach

  2011cited by this paper
• Fairness through awareness

  2011cited by this paper
• Three naive Bayes approaches for discrimination-free classification

  2010cited by this paper
• Building Classifiers with Independency Constraints

  2009cited by this paper
• Classifying without discriminating

  2009cited by this paper
• Predicting good probabilities with supervised learning

  2005cited by this paper
• Obtaining calibrated probability estimates from decision trees and naive Bayesian classifiers

  2001cited by this paper
• Probabilistic Outputs for Support vector Machines and Comparisons to Regularized Likelihood Methods

  1999cited by this paper
• The Well-Calibrated Bayesian

  1982cited by this paper

• Fix Representation (Optimally) Before Fairness: Finite-Sample Shrinkage Population Correction and the True Price of Fairness Under Subpopulation Shift

  2026cites this paper
• Exploring Fine-Tuning for Tabular Foundation Models

  2026cites this paper
• When the Server Steps In: Calibrated Updates for Fair Federated Learning

  2026cites this paper
• Bias in AI systems: integrating formal and socio-technical approaches

  2026cites this paper
• Benchmarking Bias Mitigation Toward Fairness Without Harm from Vision to LVLMs

  2026cites this paper
• Prune bias from the root: Bias removal and fairness estimation by pruning sensitive attributes in pre-trained DNN models

  2026cites this paper
• Fairness Aware Reward Optimization

  2026cites this paper
• On the Robustness of Fairness Practices: A Causal Framework for Systematic Evaluation

  2026influential citation
• The Voice of Equity: A Systematic Evaluation of Bias Mitigation Techniques for Speech-Based Cognitive Impairment Detection Across Architectures and Demographics

  2026cites this paper
• Fairness Under Group-Conditional Prior Probability Shift: Invariance, Drift, and Target-Aware Post-Processing

  2026cites this paper
• From Statistical Disclosure Control to Fair AI: Navigating Fundamental Tradeoffs in Differential Privacy

  2026cites this paper
• Separation-Utility Pareto Frontier: An Information-Theoretic Characterization

  2026cites this paper
• Failing on Bias Mitigation: Investigating Why Predictive Models Struggle with Government Data

  2026cites this paper
• Trade-offs between fairness and performance in educational AI: Analyzing post-processing bias mitigation on the OULAD

  2026cites this paper
• Bi-directional Bias Attribution: Debiasing Large Language Models without Modifying Prompts

  2026cites this paper
• Operationalizing Fairness: Post-Hoc Threshold Optimization Under Hard Resource Limits

  2026influential citation
• An Accounting Identity for Algorithmic Fairness

  2026cites this paper
• Navigating Fairness in Healthcare: A Comparative Analysis of Single-Stage and Multi-Stage Bias Mitigation Strategies

  2025cites this paper
• A Generative Agentic AI Framework for Analysis and Debiasing of Datasets and Machine Learning Models

  2025cites this paper
• AutoFairML: An Automated Middleware for Fairness Auditing in Real-world AI Pipelines

  2025cites this paper
• Toward Unifying Group Fairness Evaluation from a Sparsity Perspective

  2025influential citation
• Fair Classification by Direct Intervention on Operating Characteristics

  2025cites this paper
• Fairness-Aware Race and Ethnicity Detection From Names

  2025cites this paper
• DFCA: Disentangled Feature Contrastive Learning and Augmentation for Fairer Dermatological Diagnostics

  2025cites this paper
• DIF-PP: Threshold Optimization Informed by IRT Models for Group Fairness in Machine Learning

  2025cites this paper
• Artificial intelligence in cancer: applications, challenges, and future perspectives

  2025cites this paper
• TabTune: A Unified Library for Inference and Fine-Tuning Tabular Foundation Models

  2025cites this paper
• Evaluating algorithmic fairness of machine learning models in predicting underweight, overweight, and adiposity across socioeconomic and caste groups in India: evidence from the longitudinal ageing study in India

  2025cites this paper
• Dataset-centric AI ethics classification

  2025cites this paper
• Balancing Fairness and Accuracy in Machine Learning-Based Probability of Default Modeling via Threshold Optimization

  2025cites this paper
• Fitzpatrick Thresholding for Skin Image Segmentation

  2025cites this paper
• Happiness as a Measure of Fairness

  2025cites this paper
• Fairness for the People, by the People: Minority Collective Action

  2025cites this paper
• Understanding the Capabilities and Limitations of Weak-to-Strong Generalization

  2025cites this paper
• Revisiting Pre-processing Group Fairness: A Modular Benchmarking Framework

  2025cites this paper
• Mapping the Tradeoffs and Limitations of Algorithmic Fairness

  2025cites this paper
• Testing Fairness with Utility Tradeoffs: A Wasserstein Projection Approach

  2025cites this paper
• Leveraging Large Language Models for Bias Detection and Mitigation in Data Analytics Models

  2025cites this paper
• AI-driven healthcare: Fairness in AI healthcare: A survey

  2025cites this paper
• The Fair Game: Auditing & Debiasing AI Algorithms Over Time

  2025cites this paper
• Modeling Ranking Properties with In-Context Learning

  2025cites this paper
• Differential privacy for medical deep learning: methods, tradeoffs, and deployment implications

  2025cites this paper
• AI-powered credit risk assessment and algorithmic fairness in digital lending: A comprehensive analysis of the United States digital finance landscape

  2025cites this paper
• Machine Learning Fairness in Predicting Underweight, Overweight and Adiposity Across Socioeconomic and Caste Group in India: Evidence from the Longitudinal Ageing Study in India

  2025cites this paper
• Demystifying AI in Criminal Justice

  2025cites this paper
• Experimental Analysis of Multi-Step Pipelines for Fair Classifications - More than the Sum of Their Parts?

  2025cites this paper
• Should LLMs be WEIRD? Exploring WEIRDness and Human Rights in Large Language Models

  2025cites this paper
• Bridging fair-aware artificial intelligence and co-creation for equitable mental healthcare

  2025cites this paper
• Accuracy-fairness trade-off in ML for healthcare: A quantitative evaluation of bias mitigation strategies

  2025cites this paper
• FAIR-CARE: A comparative evaluation of unfairness mitigation approaches

  2025cites this paper
• Applications and Challenges of Fairness APIs in Machine Learning Software

  2025influential citation
• Beyond Ethics: How Inclusive Innovation Drives Economic Returns in Medical AI

  2025cites this paper
• Preference Measurement Error, Concentration in Recommendation Systems, and Persuasion

  2025cites this paper
• FairACE: Achieving Degree Fairness in Graph Neural Networks via Contrastive and Adversarial Group-Balanced Training

  2025cites this paper
• Trustworthiness Optimisation Process: A Methodology for Assessing and Enhancing Trust in AI Systems

  2025cites this paper
• Are We Aligned? A Preliminary Investigation of the Alignment of Responsible AI Values Between LLMs and Human Judgment

  2025cites this paper
• Decoupling Bias, Aligning Distributions: Synergistic Fairness Optimization for Deepfake Detection

  2025cites this paper
• Optimal Fairness under Local Differential Privacy

  2025cites this paper
• High dimensional online calibration in polynomial time

  2025cites this paper
• Why Algorithms Need Temporal Fairness

  2025cites this paper
• The future of algorithmic nondiscrimination compliance in the affordable care act

  2025cites this paper
• Metanetworks as Regulatory Operators: Learning to Edit for Requirement Compliance

  2025cites this paper
• Fairness-Aware Low-Rank Adaptation Under Demographic Privacy Constraints

  2025cites this paper
• Impact on bias mitigation algorithms to variations in inferred sensitive attribute uncertainty

  2025cites this paper
• Empirical Comparison of Post-processing Debiasing Methods for Machine Learning Classifiers in Healthcare

  2025cites this paper
• Software Engineering Principles for Fairer Systems: Experiments with GroupCART

  2025cites this paper
• Talking About the Assumption in the Room

  2025cites this paper
• A Critical Review of Predominant Bias in Neural Networks

  2025cites this paper
• Fair Foundation Models for Medical Image Analysis: Challenges and Perspectives

  2025cites this paper
• I-trustworthy Models. A framework for trustworthiness evaluation of probabilistic classifiers

  2025cites this paper
• Controlled Model Debiasing through Minimal and Interpretable Updates

  2025cites this paper
• Data-driven justice: Survey on approaches to identify and mitigate biases to build fair artificial intelligent models for Indian justice system

  2025cites this paper
• Controllable Pareto Trade-off between Fairness and Accuracy

  2025cites this paper
• AIM-Fair: Advancing Algorithmic Fairness via Selectively Fine-Tuning Biased Models with Contextual Synthetic Data

  2025cites this paper
• When and how biases seep in: Enhancing debiasing approaches for fair educational predictive analytics

  2025cites this paper
• State-of-the-Art in Responsible, Explainable, and Fair AI for Medical Image Analysis

  2025cites this paper
• Exploring trade-offs in equitable stroke risk prediction with parity-constrained and race-free models

  2025cites this paper
• seeBias: A Comprehensive Tool for Assessing and Visualizing AI Fairness

  2025cites this paper
• Whence Is A Model Fair? Fixing Fairness Bugs via Propensity Score Matching

  2025cites this paper
• Decision-centric fairness: Evaluation and optimization for resource allocation problems

  2025cites this paper
• A DIF-Driven Threshold Tuning Method for Improving Group Fairness

  2025cites this paper
• "All that Glitters": Techniques for Evaluations with Unreliable Model and Human Annotations

  2025cites this paper
• Fair Supervised Learning Through Constraints on Smooth Nonconvex Unfairness-Measure Surrogates

  2025cites this paper
• Multivariate Latent Recalibration for Conditional Normalizing Flows

  2025cites this paper
• Transparency and Proportionality in Post-Processing Algorithmic Bias Correction

  2025cites this paper
• Fairness-in-the-Workflow: How Machine Learning Practitioners at Big Tech Companies Approach Fairness in Recommender Systems

  2025cites this paper
• Algorithmic Fairness: Not a Purely Technical but Socio-Technical Property

  2025cites this paper
• Portable fair decision making through modular approach

  2025cites this paper
• HIFI: Explaining and Mitigating Algorithmic Bias Through the Lens of Game-Theoretic Interactions

  2025cites this paper
• H-Calibration: Rethinking Classifier Recalibration With Probabilistic Error-Bounded Objective

  2025cites this paper
• Balancing race, gender, and age accuracy in face recognition using DB-BETA-VAE

  2025cites this paper
• Algorithmic fairness with monotone likelihood ratios

  2025cites this paper
• ActivFairNet: A novel framework for mitigating bias in deep learning networks using activation map-based fairness regularization

  2025cites this paper
• Taming the Triangle: On the Interplays Between Fairness, Interpretability, and Privacy in Machine Learning

  2025cites this paper
• Privacy and Fairness in Machine Learning: A Survey

  2025cites this paper
• Representation-based fairness evaluation and bias correction robustness assessment in neural networks

  2025cites this paper
• Risk-averse Fair Multi-class Classification

  2025cites this paper
• 'Rich Dad, Poor Lad': How do Large Language Models Contextualize Socioeconomic Factors in College Admission ?

  2025cites this paper
• Fine-Tuning a Biased Model for Improving Fairness

  2025cites this paper
• The Capabilities and Limitations of Weak-to-Strong Generalization: Generalization and Calibration

  2025cites this paper