Fetching the paper…
Reading the bibliography…
Current practices regarding data collection for natural language processing on Amazon Mechanical Turk (MTurk) often rely on a combination of studies on data quality and heuristics shared among NLP researchers.
Cheap and fast – but is it good? evaluating non-expert annotations for natural language tasks
Rion Snow, Brendan O’Connor, Daniel Jurafsky, and Andrew Ng. 2008 · 2008
Earlier work this paper cites.
Creating speech and language data with Amazon’s Mechanical Turk
Chris Callison-Burch and Mark Dredze. 2010 · 2010
Earlier work this paper cites.
The art of question phrasing
Lior Gideon. 2012 · 2012
Earlier work this paper cites.
Mechanical Turk is not anonymous
Matthew Lease, Jessica Hullman, Jeffrey Bigham, Michael Bernstein, Juho Kim, Walter Lasecki, Saeideh Bakhshi, Tanushree Mitra, and Robert Miller. 2013 · 2013
Earlier work this paper cites.
Reputation as a sufficient condition for data quality on Amazon Mechanical Turk
Eyal Peer, Joachim Vosgerau, and Alessandro Acquisti. 2014 · 2014
Earlier work this paper cites.
Guildeines for academic requesters
Turkopticon. 2014 · 2014
Earlier work this paper cites.
The importance of assessing clinical phenomena in Mechanical Turk research
Kimberly A Arditte, Demet Çek, Ashley M Shaw, and Kiara R Timpano. 2016 · 2016
Earlier work this paper cites.
Rethinking and updating demographic questions: Guidance to improve descriptions of research samples
Jennifer Hughes, Abigail Camden, and Tenzin Yangchen. 2016 · 2016
Earlier work this paper cites.
Measuring the quality of annotations for a subjective crowdsourcing task
Raquel Justo, M. Torres, and José Alcaide. 2017 · 2017
Earlier work this paper cites.
"Our privacy needs to be protected at all costs": Crowd workers’ privacy experiences on Amazon Mechanical Turk
Huichuan Xia, Yang Wang, Yun Huang, and Anuj Shah. 2017 · 2017
Earlier work this paper cites.
A data-driven analysis of workers’ earnings on Amazon Mechanical Turk
Kotaro Hara, Abigail Adams, Kristy Milland, Saiph Savage, Chris Callison-Burch, and Jeffrey P. Bigham. 2018 · 2018
Earlier work this paper cites.
Ghost work how to stop Silicon Valley from building a new global underclass
Mary L. Gray and Siddharth Suri. 2019 · 2019
Earlier work this paper cites.
Crowdworker economics in the gig economy
Jason T. Jacques and Per Ola Kristensson. 2019 · 2019
Cited alongside, same era.
Inherent disagreements in human textual inferences
Ellie Pavlick and Tom Kwiatkowski. 2019 · 2019
Cited alongside, same era.
The risk of racial bias in hate speech detection
Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019 · 2019
Cited alongside, same era.
A question for workers with approval rate “greater than 99%”
u/Lushangdewww. 2019 · 2019
Cited alongside, same era.
Fair Work: Crowd Work Minimum Wage with One Line of Code , volume 7
Mark E. Whiting, Grant Hugh, and Michael S. Bernstein. 2019 · 2019
Cited alongside, same era.
Social bias frames: Reasoning about social and power implications of language
Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A. Smith, and Yejin Choi. 2020 · 2020
Towards automatic generation of messages countering online hate speech and microaggressions
Mana Ashida and Mamoru Komachi. 2022 · 2022
Later among the works it cites.
Inside Facebook’s African sweatshop
Billy Perrigo. 2022 · 2022
Later among the works it cites.
End the harm of mass rejections
Turkopticon. 2022 · 2022
Later among the works it cites.
Getting to the point you can’t trust any requestor paying less than a $1 (surveys)
u/Bermin299. 2022 · 2022
Later among the works it cites.
How seriously do you answer questions?
u/dgrochester55. 2022 · 2022
Later among the works it cites.
Gotta love the transparency of a seemingly reasonable requester!
u/gturker. 2022 · 2022
Later among the works it cites.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Cited alongside, same era.
Measurement and fairness
Abigail Z. Jacobs and Hanna Wallach. 2021 · 2021
Cited alongside, same era.
The perils of using Mechanical Turk to evaluate open-ended text generation
Marzena Karpinska, Nader Akoury, and Mohit Iyyer. 2021 · 2021
Cited alongside, same era.
Perceived protectiveness of research safeguards and influences on willingness to participate in research: A novel mturk pilot study
Jane Paik Kim, Katie Ryan, Tenzin Tsungmey, Max Kasun, Willa A. Roberts, Laura B. Dunn, and Laura Weiss Roberts. 2021 · 2021
Cited alongside, same era.
Quantifying and avoiding unfair qualification labour in crowdsourcing
Jonathan K. Kummerfeld. 2021 · 2021
Cited alongside, same era.
Beyond fair pay: Ethical implications of NLP crowdsourcing
Boaz Shmueli, Jan Fell, Soumya Ray, and Lun-Wei Ku. 2021 · 2021
Cited alongside, same era.
Gotta love the transparency of a seemingly reasonable requester!
u/LaughingAllTheWay83. 2022a
Cited in the paper.
Rejection from simran g
u/ptethesen. 2022 · 2022
Later among the works it cites.
Requesting ma speakers to read out loud for about 7 minutes– what is ethical pay?
u/Sharpsilverz. 2022 · 2022
Later among the works it cites.
Needle in a haystack: An analysis of finding qualified workers on mturk for summarization
Lining Zhang, João Sedoc, Simon Mille, Yufang Hou, Sebastian Gehrmann, Daniel Deutsch, Elizabeth Clark, Yixin Liu, Miruna Clinciu, Saad Mahamood, and Khyathi Chandu. 2022 · 2022
Later among the works it cites.
ChatGPT outperforms crowd-workers for text-annotation tasks
Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023 · 2023
Closest in time.
Veniamin Veselovsky, Manoel Horta Ribeiro, and Robert West. 2023 · 2023
Closest in time.