Who Said What: Modeling Individual Labelers Improves Classification

Understand

Data are often labeled by many different experts with each expert only labeling a small fraction of the data and each data point being labeled by several experts.

• This reduces the workload on individual experts and also gives a better estimate of the unobserved ground truth.
• When experts disagree, the standard approaches are to treat the majority opinion as the correct label or to model the correct label as a distribution.
• These approaches, however, do not make any use of potentially valuable information about which expert produced which label.