Accurate Uncertainties for Deep Learning Using Calibrated Regression

Understand

Methods for reasoning under uncertainty are a key building block of accurate and reliable machine learning systems.

• Bayesian methods provide a general framework to quantify uncertainty.
• However, because of model misspecification and the use of approximate inference, Bayesian uncertainty estimates are often inaccurate -- for example, a 90% credible interval may not contain the true outcome 90% of the time.
• Here, we propose a simple procedure for calibrating any regression algorithm; when applied to Bayesian and probabilistic models, it is guaranteed to produce calibrated uncertainty estimates given enough data.