Moment Multicalibration for Uncertainty Estimation

Understand

We show how to achieve the notion of "multicalibration" from H\'ebert-Johnson et al.

• [2018] not just for means, but also for variances and other higher moments.
• Informally, it means that we can find regression functions which, given a data point, can make point predictions not just for the expectation of its label, but for higher moments of its label distribution as well-and those predictions match the true distribution quantities when averaged not just over the population as a whole, but also when averaged over an enormous number of finely defined subgroups.
• It yields a principled way to estimate the uncertainty of predictions on many different subgroups-and to diagnose potential sources of unfairness in the predictive power of features across subgroups.