Understanding the Under-Coverage Bias in Uncertainty Estimation

Understand

Estimating the data uncertainty in regression tasks is often done by learning a quantile function or a prediction interval of the true label conditioned on the input.

• It is frequently observed that quantile regression -- a vanilla algorithm for learning quantiles with asymptotic guarantees -- tends to \emph{under-cover} than the desired coverage level in reality.
• While various fixes have been proposed, a more fundamental understanding of why this under-coverage bias happens in the first place remains elusive.
• In this paper, we present a rigorous theoretical study on the coverage of uncertainty estimation algorithms in learning quantiles.