Characterizing Sources of Uncertainty to Proxy Calibration and Disambiguate Annotator and Data Bias

Understand

Supporting model interpretability for complex phenomena where annotators can legitimately disagree, such as emotion recognition, is a challenging machine learning task.

• In this work, we show that explicitly quantifying the uncertainty in such settings has interpretability benefits.
• We use a simple modification of a classical network inference using Monte Carlo dropout to give measures of epistemic and aleatoric uncertainty.
• We identify a significant correlation between aleatoric uncertainty and human annotator disagreement ($r\approx.3$).