Evaluating model calibration in classification

Understand

Probabilistic classifiers output a probability distribution on target classes rather than just a class prediction.

• Besides providing a clear separation of prediction and decision making, the main advantage of probabilistic models is their ability to represent uncertainty about predictions.
• In safety-critical applications, it is pivotal for a model to possess an adequate sense of uncertainty, which for probabilistic classifiers translates into outputting probability distributions that are consistent with the empirical frequencies observed from realized outcomes.
• A classifier with such a property is called calibrated.