ActiveLab: Active Learning with Re-Labeling by Multiple Annotators

Understand

In real-world data labeling applications, annotators often provide imperfect labels.

• It is thus common to employ multiple annotators to label data with some overlap between their examples.
• We study active learning in such settings, aiming to train an accurate classifier by collecting a dataset with the fewest total annotations.
• Here we propose ActiveLab, a practical method to decide what to label next that works with any classifier model and can be used in pool-based batch active learning with one or multiple annotators.