Combining prediction intervals in the M4 competition

Abstract The 2018 M4 Forecasting Competition was the first M Competition to elicit prediction intervals in addition to point estimates. We take a closer look at the twenty valid interval submissions by examining the calibration and accuracy of the prediction intervals and evaluating their performances over different time horizons. Overall, the submissions fail to estimate the uncertainty properly. Importantly, we investigate the benefits of interval combination using six recently-proposed heuristics that can be applied prior to learning about the realizations of the quantities. Our results suggest that interval aggregation offers improvements in terms of both calibration and accuracy. While averaging interval endpoints maintains its practical appeal as being simple to implement and performs quite well when data sets are large, the median and the interior trimmed average are found to be robust aggregators for the prediction interval submissions across all 100,000 time series.

• Publication year

  2020

• Venue

  International Journal of Forecasting

• Publication date

  2020-01-01

• Fields of study

  Mathematics, Business, Computer Science

• Identifiers
  DOI 10.1016/J.IJFORECAST.2019.04.015
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• FFORMA: Feature-based forecast model averaging

  2020cited by this paper
• The M4 Competition: Results, findings, conclusion and way forward

  2018cited by this paper
• Confidence Calibration in a Multiyear Geopolitical Forecasting Competition

  2017cited by this paper
• Combining Interval Forecasts

  2017cited by this paper
• Ensembles of Overfit and Overconfident Forecasts

  2017cited by this paper
• Probabilistic energy forecasting: Global Energy Forecasting Competition 2014 and beyond

  2016cited by this paper
• Aggregating multiple probability intervals to improve calibration

  2015cited by this paper
• Trimmed Opinion Pools and the Crowd's Calibration Problem

  2014cited by this paper
• Median Aggregation of Distribution Functions

  2013cited by this paper
• Is It Better to Average Probabilities or Quantiles?

  2013cited by this paper
• Comment on Cooke's classical method

  2008cited by this paper
• The trouble with overconfidence.

  2008cited by this paper
• Advances in Decision Analysis: Aggregating Probability Distributions

  2007cited by this paper
• Strictly Proper Scoring Rules, Prediction, and Estimation

  2007cited by this paper
• Does past volatility affect investors' price forecasts and confidence judgements?

  2007cited by this paper
• Intuitions About Combining Opinions: Misappreciation of the Averaging Principle

  2006cited by this paper
• Overconfidence in interval estimates.

  2004cited by this paper
• Probability Judgments for Continuous Quantities: Linear Combinations and Calibration

  2004cited by this paper
• The M3-Competition: results, conclusions and implications

  2000cited by this paper
• Calculating Interval Forecasts

  1993cited by this paper
• Experts in Uncertainty: Opinion and Subjective Probability in Science

  1991cited by this paper
• Combining forecasts: A review and annotated bibliography

  1989cited by this paper
• Confidence intervals: An empirical investigation of the series in the M-competition

  1987cited by this paper
• Combining Probability Distributions: A Critique and an Annotated Bibliography

  1986cited by this paper
• Evidential impact of base rates

  1981cited by this paper
• The Opinion Pool

  1961cited by this paper

• Enhancing Mortality Forecasting with Ensemble Learning: A Shapley-Based Approach

  2026influential citation
• Forecasting Average Daily and Peak Electrical Load Based on Average Monthly Electricity Consumption Data

  2025cites this paper
• Probabilistic forecast aggregation with statistical depth

  2025cites this paper
• Enhancing Reliability of Electricity Supply of City Electric Networks Cities

  2024cites this paper
• Application of Deep Learning Algorithm for Prediction of Flood Severity

  2024cites this paper
• Improve ROI with Causal Learning and Conformal Prediction

  2024cites this paper
• Assessing a Fuel Subsidy: Dynamic Effects on Retailer Pricing and Pass-Through to Consumers

  2024cites this paper
• Enhancing the reliability of probabilistic PV power forecasts using conformal prediction

  2024cites this paper
• Conformal Prediction for Stochastic Decision-Making of PV Power in Electricity Markets

  2024cites this paper
• Postprocessing of point predictions for probabilistic forecasting of day-ahead electricity prices: The benefits of using isotonic distributional regression

  2024cites this paper
• Ensemble interval forecasts of mortality

  2024influential citation
• Discriminant Validity of Interval Response Formats: Investigating the Dimensional Structure of Interval Widths

  2024cites this paper
• Control of Operational Modes of an Urban Distribution Grid under Conditions of Uncertainty

  2023cites this paper
• Probabilistic Wind Power Forecasting Using Optimized Deep Auto-Regressive Recurrent Neural Networks

  2023cites this paper
• Development and evaluation of probabilistic forecasting methods for small area populations

  2023cites this paper
• DYNAMIC RESOURCE ALLOCATION ENERGY-EFFICIENT FRAMEWORK FOR GREEN CLOUD COMPUTING

  2023cites this paper
• Method for forecasting and calculating the electrical load of municipal consumers under uncertainty

  2023cites this paper
• Forecast combinations: An over 50-year review

  2022cites this paper
• A review of predictive uncertainty estimation with machine learning

  2022cites this paper
• Interpretable Battery Cycle Life Range Prediction Using Early Cell Degradation Data

  2022cites this paper
• On the Uncertainty of a Combined Forecast: The Critical Role of Correlation

  2022cites this paper
• Analysis of prognosticate Omicron Using SVM & LASSO

  2022cites this paper
• A WT-LUBE-PSO-CWC Wind Power Probabilistic Forecasting Model for Prediction Interval Construction and Seasonality Analysis

  2021cites this paper
• Machine Learning Model for Identification of Covid-19 Future Forecasting

  2021cites this paper
• Retraction Retraction: Advanced Deep Learning Model for Future

  2021cites this paper
• Advanced Deep Learning Model for Future Forecasting of COVID-19

  2021cites this paper
• Using Long Term Integrated Average Analysis Machine Learning Prediction for Spreadable Diseases

  2021cites this paper
• Post-processing in solar forecasting: Ten overarching thinking tools

  2021cites this paper
• Forecasting the Next Pandemic: Covid19 using Machine Learning

  2021cites this paper
• Harnessing the Wisdom of the Crowd to Forecast Incident and Cumulative COVID-19 Mortality in the United States

  2021cites this paper
• Covid-19 Risk Prediction Using Machine Learning Techniques

  2021cites this paper
• A data-driven interval forecasting model for building energy prediction using attention-based LSTM and fuzzy information granulation

  2021cites this paper
• Forecasting the spread of COVID-19 using supervised machine learning models

  2021cites this paper
• Combining probabilistic forecasts of COVID-19 mortality in the United States

  2021cites this paper
• A Comparison of Aggregation Methods for Probabilistic Forecasts of COVID-19 Mortality in the United States

  2020cites this paper
• Responses to discussions and commentaries

  2020cites this paper
• COVID-19 Future Forecasting Using Supervised Machine Learning Models

  2020cites this paper
• Generalizing the Theta method for automatic forecasting

  2020cites this paper
• The M4 competition: Bigger. Stronger. Better

  2020cites this paper
• Treating and Pruning: New approaches to forecasting model selection and combination using prediction intervals

  2020cites this paper
• How to model European electricity load profiles using artificial neural networks

  2020cites this paper
• Forecasting: theory and practice

  2020cites this paper