Properties of the ENCE and other MAD-based calibration metrics

Understand

The Expected Normalized Calibration Error (ENCE) is a popular calibration statistic used in Machine Learning to assess the quality of prediction uncertainties for regression problems.

• Estimation of the ENCE is based on the binning of calibration data.
• In this short note, I illustrate an annoying property of the ENCE, i.e.
• its proportionality to the square root of the number of bins for well calibrated or nearly calibrated datasets.