Principled Evaluation with Human Labels: One Rater at a Time and Rater Equivalence

Understand

In many classification tasks, there is no definitive ground truth, only human judgments that may disagree.

• We address two challenges that arise in such settings: (1) how to use human raters to score classifiers, and (2) how to use them for comparison benchmarks.
• For the first, the common practice is to score classifiers against the majority vote of an evaluation panel of several human raters.
• We argue that this is not justified when either of two properties fails: objectivity or equanimity.