Information-theoretic Generalization Analysis for Expected Calibration Error

Understand

While the expected calibration error (ECE), which employs binning, is widely adopted to evaluate the calibration performance of machine learning models, theoretical understanding of its estimation bias is limited.

• In this paper, we present the first comprehensive analysis of the estimation bias in the two common binning strategies, uniform mass and uniform width binning.
• Our analysis establishes upper bounds on the bias, achieving an improved convergence rate.
• Moreover, our bounds reveal, for the first time, the optimal number of bins to minimize the estimation bias.