AroundConnect

2021

A Survey of NLP-Related Crowdsourcing HITs: what works and what does not

Huynh, Jessica, Bigham, Jeffrey, Eskenazi, Maxine

Understand

Crowdsourcing requesters on Amazon Mechanical Turk (AMT) have raised questions about the reliability of the workers.

  • The AMT workforce is very diverse and it is not possible to make blanket assumptions about them as a group.
  • Some requesters now reject work en mass when they do not get the results they expect.
  • This has the effect of giving each worker (good or bad) a lower Human Intelligence Task (HIT) approval score, which is unfair to the good workers.

Built on

  • Financial incentives and the” performance of crowds”

    Winter Mason and Duncan J Watts. 2009 · 2009

    Earlier work this paper cites.

  • Running experiments on amazon mechanical turk

    Gabriele Paolacci, Jesse Chandler, and Panagiotis G Ipeirotis. 2010 · 2010

    Earlier work this paper cites.

  • Turkopticon: Interrupting worker invisibility in amazon mechanical turk

    Lilly C Irani and M Six Silberman. 2013 · 2013

    Earlier work this paper cites.

Similar

  • Amazon’s mechanical turk: A new source of inexpensive, yet high-quality data?

    Michael Buhrmester, Tracy Kwang, and Samuel D Gosling. 2016 · 2016

    Cited alongside, same era.

  • Taking a hit: Designing around rejection, mistrust, risk, and workers’ experiences in amazon mechanical turk

    Brian McInnis, Dan Cosley, Chaebong Nam, and Gilly Leshed. 2016 · 2016

    Cited alongside, same era.

  • Using amazon mechanical turk and other compensated crowdsourcing sites

    Gordon B Schmidt and William M Jettinghoff. 2016 · 2016

    Cited alongside, same era.

Then

  • A data-driven analysis of workers’ earnings on amazon mechanical turk

    Kotaro Hara, Abigail Adams, Kristy Milland, Saiph Savage, Chris Callison-Burch, and Jeffrey P Bigham. 2018 · 2018

    Later among the works it cites.

  • How do crowdworker communities and microtask markets influence each other? a data-driven study on amazon mechanical turk

    Jie Yang, Carlo van der Valk, Tobias Hoßfeld, Judith Redi, and Alessandro Bozzon. 2018 · 2018

    Later among the works it cites.

  • The expertise involved in deciding which hits are worth doing on amazon mechanical turk

    Benjamin V Hanrahan, Anita Chen, JiaHua Ma, Ning F Ma, Anna Squicciarini, and Saiph Savage. 2021 · 2021

    Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…