Validation of uncertainty quantification metrics: a primer based on the consistency and adaptivity concepts

Understand

The practice of uncertainty quantification (UQ) validation, notably in machine learning for the physico-chemical sciences, rests on several graphical methods (scattering plots, calibration curves, reliability diagrams and confidence curves) which explore complementary aspects of calibration, without covering all the desirable ones.

• For instance, none of these methods deals with the reliability of UQ metrics across the range of input features (adaptivity).
• Based on the complementary concepts of consistency and adaptivity, the toolbox of common validation methods for variance- and intervals- based UQ metrics is revisited with the aim to provide a better grasp on their capabilities.
• This study is conceived as an introduction to UQ validation, and all methods are derived from a few basic rules.