Annotation Imputation to Individualize Predictions: Initial Studies on Distribution Dynamics and Model Predictions

Understand

Annotating data via crowdsourcing is time-consuming and expensive.

• Due to these costs, dataset creators often have each annotator label only a small subset of the data.
• This leads to sparse datasets with examples that are marked by few annotators.
• The downside of this process is that if an annotator doesn't get to label a particular example, their perspective on it is missed.