Calibration tests in multi-class classification: A unifying framework

Understand

In safety-critical applications a probabilistic model is usually required to be calibrated, i.e., to capture the uncertainty of its predictions accurately.

• In multi-class classification, calibration of the most confident predictions only is often not sufficient.
• We propose and study calibration measures for multi-class classification that generalize existing measures such as the expected calibration error, the maximum calibration error, and the maximum mean calibration error.
• We propose and evaluate empirically different consistent and unbiased estimators for a specific class of measures based on matrix-valued kernels.