Fetching the paper…

Reading the bibliography…

2022

Metrics of calibration for probabilistic predictions

Arrieta-Ibarra, Imanol, Gujral, Paman, Tannen, Jonathan et al.

Original
PDF
Post on X

Understand

Predictions are often probabilities; e.g., a prediction could be for precipitation tomorrow, but with only a 30% chance.

Given such probabilistic predictions together with the actual outcomes, "reliability diagrams" help detect and diagnose statistically significant discrepancies -- so-called "miscalibration" -- between the predictions and the outcomes.
The canonical reliability diagrams histogram the observed and expected values of the predictions; replacing the hard histogram binning with soft kernel density estimation is another common practice.
But, which widths of bins or kernels are best? Plots of the cumulative differences between the observed and expected values largely avoid this question, by displaying miscalibration directly as the slopes of secant lines for the graphs.

Metrics of calibration for probabilistic predictions · Around

Built on

A. N. Kolmogorov
1933
Earlier work this paper cites.
N. Smirnov
1939
Earlier work this paper cites.
N. H. Kuiper
1962
Earlier work this paper cites.
J. S. Simonoff and F. Udina
1997
Earlier work this paper cites.
W. Stute
1997
Earlier work this paper cites.
Also published as a poster and paper at the 2021 International Conference on Learned Representations (ICLR). Available at https://arxiv.org/abs/2006.12800
Original
K. Gupta, A. Rahimi, T. Ajanthan, T. Mensink, C. Sminchisescu, and R. Hartley · 2006
Earlier work this paper cites.

Similar

J. Bröcker
2008
Cited alongside, same era.
Available at https://arxiv.org/abs/2008.01779
Original
M. Tygert · 2008
Cited alongside, same era.
Available at https://arxiv.org/abs/2012.08668
Original
R. Roelofs, N. Cain, J. Shlens, and M. C. Mozer · 2012
Cited alongside, same era.
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei
2015
Cited alongside, same era.
K. He, X. Zhang, S. Ren, and J. Sun
2016
Cited alongside, same era.
A. Kumar, P. Liang, and T. Ma
2019
Cited alongside, same era.

Then

J. Nixon, M. W. Dusenberry, L. Zhang, G. Jerfel, and D. Tran
2019
Later among the works it cites.
A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala
2019
Later among the works it cites.
Proc. 22nd Int. Conf. Artif. Intell. Stat
J. Vaicenavicius, D. Widmann, C. Andersson, F. Lindsten, J. Roll, and T. B. Schön · 2019
Later among the works it cites.
Available at https://arxiv.org/abs/2108.02666
Original
2021
Later among the works it cites.
Available at https://arxiv.org/abs/2202.00100
Original
2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

R. Srihera and W. Stute

2059

Closest in time.