Exact Exponent in Optimal Rates for Crowdsourcing

Understand

In many machine learning applications, crowdsourcing has become the primary means for label collection.

• In this paper, we study the optimal error rate for aggregating labels provided by a set of non-expert workers.
• Under the classic Dawid-Skene model, we establish matching upper and lower bounds with an exact exponent $mI(\pi)$ in which $m$ is the number of workers and $I(\pi)$ the average Chernoff information that characterizes the workers' collective ability.
• Such an exact characterization of the error exponent allows us to state a precise sample size requirement $m>\frac{1}{I(\pi)}\log\frac{1}{\epsilon}$ in order to achieve an $\epsilon$ misclassification error.