Rashomon Capacity: A Metric for Predictive Multiplicity in Classification

Understand

Predictive multiplicity occurs when classification models with statistically indistinguishable performances assign conflicting predictions to individual samples.

• When used for decision-making in applications of consequence (e.g., lending, education, criminal justice), models developed without regard for predictive multiplicity may result in unjustified and arbitrary decisions for specific individuals.
• We introduce a new metric, called Rashomon Capacity, to measure predictive multiplicity in probabilistic classification.
• Prior metrics for predictive multiplicity focus on classifiers that output thresholded (i.e., 0-1) predicted classes.