Information-Theoretic Measures of Metacognition: Bounds and Relation to Group Performance

Abstract Metacognition comprises the ability to differentiate the accuracy of predictions about the world. This is often called Type 2 performance (with Type 1 performance being the overall accuracy). Typical measures of metacognition are based on signal detection theory and require the strong assumption of truncated normal noise underlying confidence ratings. To minimize distributional assumptions, measures based on classical information theory have been proposed. We further this approach by providing bounds on its key quantity, the transmitted information. We show that classifiers making predictions with a certain accuracy can transmit information only within a limited range, depending on the underlying noise distribution: The lowest transmitted information indicates the worst Type 2 performance and corresponds to binary noise; the highest transmitted information indicates the best Type 2 performance and corresponds to uniform noise. Because normal noise is only an intermediate case, traditional measures based on this assumption can bias interpretations of Type 2 performance. Based on these bounds, we suggest a new measure: Relative metainformation (RMI). RMI scales from 0 (lower bound) to 1 (upper bound) and therefore advances towards the much-needed decoupling of Type 2 from Type 1 performance measures. To demonstrate the strengths of RMI, we apply it to groups: In a setting where multiple independent group members with fixed accuracies combine their predictions in an optimal way, we show that the group performance depends directly on RMI: Group accuracy is best vs. worst if the group members have highest vs. lowest RMI values. Overall, our theoretical bounds allow to better evaluate measures of Type 2 and group performance.

• Publication year

  2025

• Venue

  Open Mind

• Publication date

  2025-10-17

• Fields of study

  Mathematics, Computer Science, Medicine

• Identifiers
  DOI 10.1162/OPMI.a.40PMID 41246215PMCID 12618015
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar, PubMed

• No claims are published for this paper.

• No concepts are published for this paper.

• Automatic multisensory integration follows subjective confidence rather than objective performance

  2025cited by this paper
• A comprehensive assessment of current methods for measuring metacognition

  2025cited by this paper
• Collective Intelligence Increases Diagnostic Accuracy in a General Practice Setting

  2024cited by this paper
• Does Using None-of-the-Above (NOTA) Hurt Students’ Confidence?

  2023cited by this paper
• Measures of metacognitive efficiency across cognitive models of decision confidence.

  2023cited by this paper
• How do humans give confidence? A comprehensive comparison of process models of perceptual metacognition.

  2023cited by this paper
• On the Relationship between Feature Selection Metrics and Accuracy

  2023cited by this paper
• The future of metacognition research: Balancing construct breadth with measurement rigor

  2023cited by this paper
• Metacognition and Confidence: A Review and Synthesis.

  2023cited by this paper
• Metacognitive Information Theory

  2023influential reference
• Supplemental Material for Harnessing the Wisdom of the Confident Crowd in Medical Image Decision-Making

  2023cited by this paper
• Optimisation & Generalisation in Networks of Neurons

  2022cited by this paper
• Taking a Step Back with KCal: Multi-Class Kernel-Based Calibration for Deep Neural Networks

  2022cited by this paper
• An application of simple majority rule to a group with an even number of voters

  2022cited by this paper
• Deep problems with neural network models of human vision

  2022cited by this paper
• Confidence reflects a noisy decision reliability estimate

  2022cited by this paper
• Model-free metacognition.

  2022cited by this paper
• On Margins and Generalisation for Voting Classifiers

  2022cited by this paper
• Stop Measuring Calibration When Humans Disagree

  2022cited by this paper
• A Consistent and Efficient Evaluation Strategy for Attribution Methods

  2022cited by this paper
• The value of confidence: Confidence prediction errors drive value-based learning in the absence of external feedback

  2021cited by this paper
• Kernel Interpolation as a Bayes Point Machine

  2021cited by this paper
• Measuring metacognitive performance: type 1 performance dependence and test-retest reliability

  2021cited by this paper
• Reverse engineering of metacognition

  2021cited by this paper
• Mutual Information and Categorical Perception

  2021cited by this paper
• Information-theoretic signal detection theory.

  2021cited by this paper
• A survey of uncertainty in deep neural networks

  2021cited by this paper
• Ensemble deep learning: A review

  2021cited by this paper
• Metacognitive Computations for Information Search: Confidence in Control

  2021cited by this paper
• Maximum-relevance and maximum-diversity of positive ranks: A novel feature selection method

  2020cited by this paper
• The forgotten history of signal detection theory.

  2020cited by this paper
• Confounding in Studies on Metacognition: A Preliminary Causal Analysis Framework

  2020cited by this paper
• Dynamic influences on static measures of metacognition

  2020cited by this paper
• The nature of metacognitive inefficiency in perceptual decision making.

  2020cited by this paper
• Group decisions based on confidence weighted majority voting

  2020cited by this paper
• The bounded rationality of probability distortion

  2019cited by this paper
• How experimental procedures influence estimates of metacognitive ability

  2019cited by this paper
• Introducing Confidence as a Weight in Random Forest

  2019cited by this paper
• Comparative Accuracy of Diagnosis by Collective Intelligence of Multiple Physicians vs Individual Physicians

  2019cited by this paper
• Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration

  2019cited by this paper
• Forming global estimates of self-performance from local confidence

  2019cited by this paper
• Supra-optimality may emanate from suboptimality, and hence optimality is no benchmark in multisensory integration

  2018cited by this paper
• The folded X-pattern is not necessarily a statistical signature of decision confidence

  2018cited by this paper
• A Tight Upper Bound on Mutual Information

  2018cited by this paper
• On Calibration of Modern Neural Networks

  2017cited by this paper
• Making better decisions in groups

  2017cited by this paper
• Feature Selection by Maximizing Independent Classification Information

  2017cited by this paper
• Brain networks for confidence weighting and hierarchical inference during probabilistic learning

  2017cited by this paper
• Bayes-Like Integration of a New Sensory Skill with Vision

  2017cited by this paper
• Post-decisional accounts of biases in confidence

  2016cited by this paper
• Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles

  2016cited by this paper
• Boosting medical diagnostics by pooling independent judgments

  2016cited by this paper
• The Effect of Visual Appearance on the Performance of Continuous Sliders and Visual Analogue Scales

  2016cited by this paper
• Does dirichlet prior smoothing solve the Shannon entropy estimation problem?

  2015cited by this paper
• When two heads are better than one and when they can be worse: The amplification hypothesis.

  2015cited by this paper
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning

  2015cited by this paper
• Signal Detection Theory Analysis of Type 1 and Type 2 Data: Meta-d′, Response-Specific Meta-d′, and the Unequal Variance SDT Model

  2014cited by this paper
• How to measure metacognition

  2014influential reference
• Error Rate Bounds and Iterative Weighted Majority Voting for Crowdsourcing

  2014cited by this paper
• Minimax Estimation of Functionals of Discrete Distributions

  2014cited by this paper
• The subjective experience of object recognition: comparing metacognition for object detection and object categorization

  2014cited by this paper
• Identification of stimulus cues in narrow-band tone-in-noise detection using sparse observer models.

  2013cited by this paper
• A review of feature selection methods based on mutual information

  2013cited by this paper
• A weighted random forests approach to improve predictive performance

  2013cited by this paper
• Boosting: Foundations and Algorithms

  2013cited by this paper
• Measures of metacognition on signal-detection theoretic models.

  2013cited by this paper
• Refinement revisited with connections to Bayes error, conditional entropy and calibrated classifiers

  2013cited by this paper
• Consistency of weighted majority votes

  2013cited by this paper
• Bayesian entropy estimation for countable discrete distributions

  2013cited by this paper
• Ideal-Observer Models of Cue Integration

  2012cited by this paper
• Ubiquitous Log Odds: A Common Representation of Probability and Frequency Distortion in Perception, Action, and Cognition

  2012cited by this paper
• Improving Tree augmented Naive Bayes for class probability estimation

  2012cited by this paper
• A computational framework for the study of confidence in humans and animals

  2012cited by this paper
• Mechanisms of social cognition.

  2012cited by this paper
• A weighted voting framework for classifiers ensembles

  2012influential reference
• A signal detection theoretic approach for estimating metacognitive sensitivity from confidence ratings.

  2012cited by this paper
• Is specialization desirable in committee decision making?

  2012cited by this paper
• The role of metacognition in human social interactions

  2012cited by this paper
• Ensembles in Machine Learning Applications

  2011cited by this paper
• What Are the Differences Between Bayesian Classifiers and Mutual-Information Classifiers?

  2011cited by this paper
• Sensory Cue Integration

  2011cited by this paper
• Optimally Interacting Minds

  2010cited by this paper
• Subjective discriminability of invisibility: a framework for distinguishing perceptual and attentional failures of awareness.

  2010cited by this paper
• Ensemble-based classifiers

  2010cited by this paper
• Essai Sur L'Application de L'Analyse a la Probabilite Des Decisions Rendues a la Pluralite Des Voix

  2009cited by this paper
• Auditory dominance over vision in the perception of interval duration

  2009cited by this paper
• Reliability, Sufficiency, and the Decomposition of Proper Scores

  2008cited by this paper
• Decision theory, reinforcement learning, and the brain

  2008cited by this paper
• Classification confidence weighted majority voting using decision tree classifiers

  2008cited by this paper
• Estimating class probabilities in random forests

  2007cited by this paper
• An introduction to ROC analysis

  2006cited by this paper
• Elements of information theory (2. ed.)

  2006cited by this paper
• Conditional Independence Trees

  2004cited by this paper
• Bayesian Computation in Recurrent Neural Circuits

  2004cited by this paper
• The Bayesian brain: the role of uncertainty in neural coding and computation.

  2004cited by this paper
• Information Theory, Inference, and Learning Algorithms

  2004cited by this paper
• Drawing the line on metacognition.

  2003cited by this paper
• Type 2 tasks in the theory of signal detectability: Discrimination between correct and incorrect decisions

  2003cited by this paper
• Rate-Distortion Theory

  2003cited by this paper
• Estimation of Entropy and Mutual Information

  2003cited by this paper

• No citing papers are available for this paper.