Learning from Crowds with Crowd-Kit

Understand

This paper presents Crowd-Kit, a general-purpose computational quality control toolkit for crowdsourcing.

• Crowd-Kit provides efficient and convenient implementations of popular quality control algorithms in Python, including methods for truth inference, deep learning from crowds, and data quality estimation.
• Our toolkit supports multiple modalities of answers and provides dataset loaders and example notebooks for faster prototyping.
• We extensively evaluated our toolkit on several datasets of different natures, enabling benchmarking computational quality control methods in a uniform, systematic, and reproducible way using the same codebase.