CrowdTruth 2.0: Quality Metrics for Crowdsourcing with Disagreement

Understand

Typically crowdsourcing-based approaches to gather annotated data use inter-annotator agreement as a measure of quality.

• However, in many domains, there is ambiguity in the data, as well as a multitude of perspectives of the information examples.
• In this paper, we present ongoing work into the CrowdTruth metrics, that capture and interpret inter-annotator disagreement in crowdsourcing.
• The CrowdTruth metrics model the inter-dependency between the three main components of a crowdsourcing system -- worker, input data, and annotation.