Fetching the paper…
Reading the bibliography…
We introduce Mephisto, a framework to make crowdsourcing for research more reproducible, transparent, and collaborative.
ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisons
Li, M.; Weston, J.; and Roller, S. 2019 · 1909
Earlier work this paper cites.
Annotationsaurus: A Searchable Directory of Annotation Tools
Neves, M.; and Seva, J. 2020 · 2010
Earlier work this paper cites.
LabelMe: Online Image Annotation and Applications By developing a publicly available tool that allows users to use the Internet to quickly and easily annotate images, the authors were able to collect many detailed image descriptions
Torralba, A.; Russell, B. C.; and Yuen, J. 2010 · 2010
Earlier work this paper cites.
Recipes for Safety in Open-domain Chatbots
Xu, J.; Ju, D.; Li, M.; Boureau, Y.; Weston, J.; and Dinan, E. 2020 · 2010
Earlier work this paper cites.
Who Moderates the Moderators? Crowdsourcing Abuse Detection in User-Generated Content
Ghosh, A.; Kale, S.; and McAfee, P. 2011 · 2011
Earlier work this paper cites.
Active learning
Settles, B. 2012 · 2012
Earlier work this paper cites.
Keep It Simple: Reward and Task Design in Crowdsourcing
Finnerty, A.; Kucherbaev, P.; Tranquillini, S.; and Convertino, G. 2013 · 2013
Earlier work this paper cites.
ShapeNet: An Information-Rich 3D Model Repository
Chang, A. X.; Funkhouser, T. A.; Guibas, L. J.; Hanrahan, P.; Huang, Q.; Li, Z.; Savarese, S.; Savva, M.; Song, S.; Su, H.; Xiao, J.; Yi, L.; and Yu, F. 2015 · 2015
Earlier work this paper cites.
psiTurk: An open-source framework for conducting replicable behavioral experiments online
Gureckis, T.; Martin, J.; McDonnell, J.; Rich, A.; Markant, D.; Coenen, A.; Halpern, D.; Hamrick, J.; and Chan, P. 2016 · 2015
Earlier work this paper cites.
Directions in Hybrid Intelligence: Complementing AI Systems with Human Intelligence
Kamar, E. 2016 · 2016
Earlier work this paper cites.
Reliable Crowdsourcing under the Generalized Dawid-Skene Model
Khetan, A.; and Oh, S. 2016 · 2016
Cited alongside, same era.
”Why Should I Trust You?”: Explaining the Predictions of Any Classifier
Ribeiro, M. T.; Singh, S.; and Guestrin, C. 2016 · 2016
Cited alongside, same era.
Scribe: Deep Integration of Human and Machine Intelligence to Caption Speech in Real Time
Lasecki, W. S.; Miller, C. D.; Naim, I.; Kushalnagar, R.; Sadilek, A.; Gildea, D.; and Bigham, J. P. 2017 · 2017
Cited alongside, same era.
ParlAI: A Dialog Research Software Platform
Miller, A. H.; Feng, W.; Fisch, A.; Lu, J.; Batra, D.; Bordes, A.; Parikh, D.; and Weston, J. 2017 · 2017
Cited alongside, same era.
Deep Learning is Robust to Massive Label Noise
Rolnick, D.; Veit, A.; Belongie, S.; and Shavit, N. 2017 · 2017
Cited alongside, same era.
Assuring the Machine Learning Lifecycle: Desiderata, Methods, and Challenges
Ashmore, R.; Calinescu, R.; and Paterson, C. 2021 · 2021
Later among the works it cites.
Ego4D: Around the World in 3, 000 Hours of Egocentric Video
Grauman, K.; Westbury, A.; Byrne, E.; Chavis, Z.; Furnari, A.; Girdhar, R.; Hamburger, J.; Jiang, H.; Liu, M.; Liu, X.; et al. 2021 · 2021
Later among the works it cites.
Towards Accountability for Machine Learning Datasets: Practices from Software Engineering and Infrastructure
Hutchinson, B.; Smart, A.; Hanna, A.; Denton, E.; Greer, C.; Kjartansson, O.; Barnes, P.; and Mitchell, M. 2021 · 2021
Later among the works it cites.
LEGOEval: An Open-Source Toolkit for Dialogue System Evaluation via Crowdsourcing
Li, Y.; Arnold, J.; Yan, F.; Shi, W.; and Yu, Z. 2021 · 2021
Later among the works it cites.
Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks
Northcutt, C. G.; Athalye, A.; and Mueller, J. 2021 · 2021
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Confusing the Crowd: Task Instruction Quality on Amazon Mechanical Turk
Wu, M.-H.; and Quinn, A. J. 2017 · 2017
Cited alongside, same era.
Sprout: Crowd-Powered Task Design for Crowdsourcing
Bragg, J.; Mausam; and Weld, D. S. 2018 · 2018
Cited alongside, same era.
Making Better Use of the Crowd: How Crowdsourcing Can Advance Machine Learning Research
Vaughan, J. W. 2018 · 2018
Cited alongside, same era.
Machine Learning with Crowdsourcing: A Brief Summary of the Past Research and Future Directions
Sheng, V. S.; and Zhang, J. 2019 · 2019
Cited alongside, same era.
Scaling Laws for Neural Language Models
Kaplan, J.; McCandlish, S.; Henighan, T.; Brown, T. B.; Chess, B.; Child, R.; Gray, S.; Radford, A.; Wu, J.; and Amodei, D. 2020 · 2020
Cited alongside, same era.
Later among the works it cites.
Data and its (dis) contents: A survey of dataset development and use in machine learning research
Paullada, A.; Raji, I. D.; Bender, E. M.; Denton, E.; and Hanna, A. 2021 · 2021
Later among the works it cites.
“Everyone Wants to Do the Model Work, Not the Data Work”: Data Cascades in High-Stakes AI
Sambasivan, N.; Kapania, S.; Highfill, H.; Akrong, D.; Paritosh, P.; and Aroyo, L. M. 2021 · 2021
Later among the works it cites.
Machine Learning: Algorithms, Real-World Applications and Research Directions
Sarker, I. 2021 · 2021
Later among the works it cites.
Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AI
Pushkarna, M.; Zaldivar, A.; and Kjartansson, O. 2022 · 2022
Later among the works it cites.
Beyond neural scaling laws: beating power law scaling via data pruning
Sorscher, B.; Geirhos, R.; Shekhar, S.; Ganguli, S.; and Morcos, A. S. 2022 · 2022
Later among the works it cites.