Learning from Imperfect Annotations

Understand

Many machine learning systems today are trained on large amounts of human-annotated data.

• Data annotation tasks that require a high level of competency make data acquisition expensive, while the resulting labels are often subjective, inconsistent, and may contain a variety of human biases.
• To improve the data quality, practitioners often need to collect multiple annotations per example and aggregate them before training models.
• Such a multi-stage approach results in redundant annotations and may often produce imperfect "ground truth" that may limit the potential of training accurate machine learning models.