A universal method for evaluating the quality of aggregators

We propose a new method to facilitate comparison of aggregated forecasts based on different aggregation, elicitation and calibration methods. Aggregates are evaluated by their relative position on the cumulative distribution of the corresponding individual scores. This allows one to compare methods using different measures of quality that use different scales. We illustrate the use of the method by re-analyzing various estimates from Budescu and Du (Management Science, 2007).

• Publication year

  2019

• Venue

  Judgment and Decision Making

• Publication date

  2019-07-01

• Fields of study

  Not labeled

• Identifiers
  DOI 10.1017/s1930297500006094
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Boosting intelligence analysts’ judgment accuracy: What works, what fails?

  2018cited by this paper
• Quantile Evaluation, Sensitivity to Bracketing, and Sharing Business Payoffs

  2017cited by this paper
• Combining Interval Forecasts

  2017cited by this paper
• Efficiently encoding and modeling subjective probability distributions for quantitative variables.

  2016cited by this paper
• Do We Measure Overconfidence? A Closer Look at the Interval Production Task

  2016cited by this paper
• Validating the Contribution-Weighted Model: Robustness and Cost-Benefit Analyses

  2016cited by this paper
• Aggregating multiple probability intervals to improve calibration

  2015cited by this paper
• Identifying Expertise to Extract the Wisdom of Crowds

  2015cited by this paper
• Forecast aggregation via recalibration

  2014cited by this paper
• When is a crowd wise?

  2014cited by this paper
• Combining multiple probability predictions using a simple logit model

  2014cited by this paper
• Two Reasons to Make Aggregated Probability Forecasts More Extreme

  2014cited by this paper
• Identifying expertise and using it to extract the Wisdom of the Crowds

  2013cited by this paper
• Choosing a Strictly Proper Scoring Rule

  2013cited by this paper
• Median Aggregation of Distribution Functions

  2013cited by this paper
• Judgment and decision making.

  2012cited by this paper
• A simple remedy for overprecision in judgment.

  2010cited by this paper
• Strategies for revising judgment: how (and how well) people use others' opinions.

  2009cited by this paper
• Evaluating Quantile Assessments

  2009cited by this paper
• Overconfidence in interval estimates: What does expertise buy you?

  2008cited by this paper
• A Comparison of Two Probability Encoding Methods: Fixed Probability vs. Fixed Variable Values

  2008cited by this paper
• Coherence and Consistency of Investors' Probability Judgments

  2007cited by this paper
• Intuitions About Combining Opinions: Misappreciation of the Averaging Principle

  2006cited by this paper
• Another look at measures of forecast accuracy

  2006cited by this paper
• When 90% confidence intervals are 50% certain: on the credibility of credible intervals

  2005cited by this paper
• Overconfidence in interval estimates.

  2004cited by this paper
• An Introduction to the Ecb's Survey of Professional Forecasters

  2003cited by this paper
• Heuristics and Biases: The Psychology of Intuitive Judgment

  2002cited by this paper
• The effects of averaging subjective probability estimates between and within judges.

  2000cited by this paper
• Format dependence in subjective probability calibration

  1999cited by this paper
• Overconfidence: It Depends on How, What, and Whom You Ask.

  1999influential reference
• Simultaneous Over- and Underconfidence: The Role of Error in Judgment Processes.

  1994cited by this paper
• Introducing: The Survey of Professional Forecasters

  1993cited by this paper
• Error Measures for Generalizing About Forecasting Methods: Empirical Comparisons

  1992cited by this paper
• A risk assessment for selected lead-induced health effects: an example of a general methodology.

  1989cited by this paper
• Judgment under uncertainty: A progress report on the training of probability assessors

  1982cited by this paper
• Knowing with Certainty: The Appropriateness of Extreme Confidence.

  1977cited by this paper
• A Progress Report

  1974cited by this paper
• A New Vector Partition of the Probability Score

  1973cited by this paper
• Function minimization by conjugate gradients

  1964cited by this paper

• Boosting the Wisdom of Crowds Within a Single Judgment Problem: Weighted Averaging Based on Peer Predictions

  2023cites this paper
• Recalibrating probabilistic forecasts to improve their accuracy

  2022cites this paper
• The Wisdom of the Coherent: Improving Correspondence with Coherence-Weighted Aggregation

  2021cites this paper
• Uncertain About Uncertainty: How Qualitative Expressions of Forecaster Confidence Impact Decision-Making With Uncertainty Visualizations

  2021cites this paper
• Mathematically aggregating experts’ predictions of possible futures

  2021cites this paper
• Quantifying machine influence over human forecasters

  2020cites this paper
• Crowd control: Reducing individual estimation bias by sharing biased social information

  2020influential citation
• Anchor Attention for Hybrid Crowd Forecasts Aggregation

  2020cites this paper