Random Noise vs State-of-the-Art Probabilistic Forecasting Methods : A Case Study on CRPS-Sum Discrimination Ability

Understand

The recent developments in the machine learning domain have enabled the development of complex multivariate probabilistic forecasting models.

• Therefore, it is pivotal to have a precise evaluation method to gauge the performance and predictability power of these complex methods.
• To do so, several evaluation metrics have been proposed in the past (such as Energy Score, Dawid-Sebastiani score, variogram score), however, they cannot reliably measure the performance of a probabilistic forecaster.
• Recently, CRPS-sum has gained a lot of prominence as a reliable metric for multivariate probabilistic forecasting.