Fetching the paper…
Reading the bibliography…
We propose a novel three-stage FIND-RESOLVE-LABEL workflow for crowdsourced annotation to reduce ambiguity in task instructions and thus improve annotation quality.
Design Patterns: Elements of Reusable Object-oriented Software (Boston, MA, USA: Addison-Wesley Longman Publishing Co., Inc.)
Gamma, E., Helm, R., Johnson, R., and Vlissides, J. (1995) · 1995
Earlier work this paper cites.
Help Find Jim Gray
Vogels, W. (2007) · 2007
Earlier work this paper cites.
Crowdsourcing for relevance evaluation
Alonso, O., Rose, D. E., and Stewart, B. (2008) · 2008
Earlier work this paper cites.
How evaluator domain expertise affects search result relevance judgments
Kinney, K. A., Huffman, S. B., and Zhai, J. (2008) · 2008
Earlier work this paper cites.
Galaxy zoo: morphologies derived from visual inspection of galaxies from the sloan digital sky survey
Lintott, C. J., Schawinski, K., Slosar, A., Land, K., Bamford, S., Thomas, D., et al. (2008) · 2008
Earlier work this paper cites.
Cheap and fast—but is it good?: evaluating non-expert annotations for natural language tasks
Snow, R., O’Connor, B., Jurafsky, D., and Ng, A. Y. (2008) · 2008
Earlier work this paper cites.
Utility data annotation with amazon mechanical turk
Sorokin, A. and Forsyth, D. (2008) · 2008
Earlier work this paper cites.
Soylent: a word processor with a crowd inside
Bernstein, M. S., Little, G., Miller, R. C., Hartmann, B., Ackerman, M. S., Karger, D. R., et al. (2010) · 2010
Earlier work this paper cites.
Crowdsourcing document relevance assessment with mechanical turk
Grady, C. and Lease, M. (2010) · 2010
Earlier work this paper cites.
Quality management on amazon mechanical turk
Ipeirotis, P. G., Provost, F., and Wang, J. (2010) · 2010
Earlier work this paper cites.
A study of the human flesh search engine: crowd-powered expansion of online knowledge
Wang, F.-Y., Zeng, D., Hendler, J. A., Zhang, Q., Feng, Z., Gao, Y., et al. (2010) · 2010
Earlier work this paper cites.
The jabberwocky programming environment for structured social computing
Ahmad, S., Battle, A., Malkani, Z., and Kamvar, S. (2011) · 2011
Earlier work this paper cites.
Beat the machine: Challenging workers to find the unknown unknowns
Attenberg, J., Ipeirotis, P. G., and Provost, F. (2011) · 2011
Earlier work this paper cites.
Opportunities for crowdsourcing research on amazon mechanical turk
Chen, J. J., Menezes, N. J., Bradley, A. D., and North, T. (2011) · 2011
Earlier work this paper cites.
A theory of information need for information retrieval that connects information to knowledge
Cole, C. (2011) · 2011
Earlier work this paper cites.
Crowddb: answering queries with crowdsourcing
Franklin, M. J., Kossmann, D., Kraska, T., Ramesh, S., and Xin, R. (2011) · 2011
Earlier work this paper cites.
Crowdsourcing for book search evaluation: impact of hit design on comparative system ranking
Kazai, G., Kamps, J., Koolen, M., and Milic-Frayling, N. (2011) · 2011
Earlier work this paper cites.
Crowdforge: Crowdsourcing complex work
Kittur, A., Smus, B., Khamkar, S., and Kraut, R. E. (2011) · 2011
Earlier work this paper cites.
Time-critical social mobilization
Pickard, G., Pan, W., Rahwan, I., Cebrian, M., Crane, R., Madan, A., et al. (2011) · 2011
Earlier work this paper cites.
Shepherding the crowd yields better work
Dow, S., Kulkarni, A., Klemmer, S., and Hartmann, B. (2012) · 2012
Earlier work this paper cites.
Crowdweaver: visually managing complex crowd work
Kittur, A., Khamkar, S., André, P., and Kraut, R. (2012) · 2012
Earlier work this paper cites.
Collaboratively crowdsourcing workflows with turkomatic
Kulkarni, A., Can, M., and Hartmann, B. (2012a) · 2012
Earlier work this paper cites.
Learning from crowds in the presence of schools of thought
Tian, Y. and Zhu, J. (2012) · 2012
Earlier work this paper cites.
An evaluation of aggregation techniques in crowdsourcing
Hung, N. Q. V., Tam, N. T., Tran, L. N., and Aberer, K. (2013) · 2013
Earlier work this paper cites.
An introduction to crowdsourcing for language and multimedia technology research
Jones, G. J. (2013) · 2013
Cited alongside, same era.
A human/computer learning network to improve biodiversity conservation and research
Kelling, S., Gerbracht, J., Fink, D., Lagoze, C., Wong, W.-K., Yu, J., et al. (2013) · 2013
Cited alongside, same era.
Experiences surveying the crowd: Reflections on methods, participation, and reliability
Marshall, C. C. and Shipman, F. M. (2013) · 2013
Cited alongside, same era.
The effect of threshold priming and need for cognition on relevance calibration and assessment
Scholer, F., Kelly, D., Wu, W.-C., Lee, H. S., and Webber, W. (2013) · 2013
Cited alongside, same era.
SQUARE: A Benchmark for Research on Computing Crowd Consensus
Sheshadri, A. and Lease, M. (2013) · 2013
Cited alongside, same era.
Crowdsourcing in hci research
Revolt: Collaborative crowdsourcing for labeling machine learning datasets
Chang, J. C., Amershi, S., and Kamar, E. (2017) · 2017
Later among the works it cites.
Clarity is a worthwhile quality: On the role of task clarity in microtask crowdsourcing
Gadiraju, U., Yang, J., and Bozzon, A. (2017) · 2017
Later among the works it cites.
The daemo crowdsourcing marketplace
Gaikwad, S. N. S., Whiting, M. E., Gamage, D., Mullings, C. A., Majeti, D., Goyal, S., et al. (2017) · 2017
Later among the works it cites.
Shifts in rating bias due to scale saturation
Kalra, K., Patwardhan, M., and Karande, S. (2017) · 2017
Later among the works it cites.
Confusing the crowd: Task instruction quality on amazon mechanical turk
Wu, M.-H. and Quinn, A. J. (2017) · 2017
Later among the works it cites.
Truth inference in crowdsourcing: is the problem solved?
Zheng, Y., Li, G., Li, Y., Shan, C., and Cheng, R. (2017) · 2017
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Egelman, S., Chi, E. H., and Dow, S. (2014) · 2014
Cited alongside, same era.
Structured labeling for facilitating concept evolution in machine learning
Kulesza, T., Amershi, S., Caruana, R., Fisher, D., and Charles, D. (2014) · 2014
Cited alongside, same era.
Expert crowdsourcing with flash teams
Retelny, D., Robaszkiewicz, S., To, A., Lasecki, W. S., Patel, J., Rahmati, N., et al. (2014) · 2014
Cited alongside, same era.
Practical lessons for gathering quality labels at scale
Alonso, O. (2015) · 2015
Cited alongside, same era.
Flock: Hybrid crowd-machine learning classifiers
Cheng, J. and Bernstein, M. S. (2015) · 2015
Cited alongside, same era.
Modeling Temporal Crowd Work Quality with Limited Supervision
Jung, H. J. and Lease, M. (2015) · 2015
Cited alongside, same era.
Crowdsourcing from scratch: A pragmatic experiment in data collection by novice requesters
Papoutsaki, A., Guo, H., Metaxa-Kakavouli, D., Gramazio, C., Rasley, J., Xie, W., et al. (2015) · 2015
Cited alongside, same era.
Later among the works it cites.
Sprout: Crowd-powered task design for crowdsourcing
Bragg, J. and Weld, D. S. (2018) · 2018
Later among the works it cites.
Crowdtruth 2.0: Quality metrics for crowdsourcing with disagreement
Dumitrache, A., Inel, O., Aroyo, L., Timmermans, B., and Welty, C. (2018) · 2018
Later among the works it cites.
Wingit: Efficient refinement of unclear task instructions
Manam, V. C. and Quinn, A. J. (2018) · 2018
Later among the works it cites.
Resolvable vs. irresolvable disagreement: A study on worker deliberation in crowd work
Schaekermann, M., Goh, J., Larson, K., and Law, E. (2018) · 2018
Later among the works it cites.
Cicero: Multi-turn, contextual argumentation for accurate crowdsourcing
Chen, Q., Bragg, J., Chilton, L. B., and Weld, D. S. (2019) · 2019
Later among the works it cites.
Are we modeling the task or the annotator? an investigation of annotator bias in natural language understanding datasets
Geva, M., Goldberg, Y., and Berant, J. (2019) · 2019
Later among the works it cites.
Taskmate: A mechanism to improve the quality of instructions in crowdsourcing
Manam, V. K. C., Jampani, D., Zaim, M., Wu, M.-H., and J. Quinn, A. (2019) · 2019
Later among the works it cites.
Crowds and camera traps: Genres in online citizen science projects
Rosser, H. and Wiggins, A. (2019) · 2019
Later among the works it cites.
A hybrid approach to identifying unknown unknowns of predictive models
Vandenhof, C. (2019) · 2019
Later among the works it cites.
Identifying and measuring annotator bias based on annotators’ demographic characteristics
Al Kuwatly, H., Wich, M., and Groh, G. (2020) · 2020
Later among the works it cites.
Detecting and preventing confused labels in crowdsourced data
Krivosheev, E., Bykau, S., Casati, F., and Prabhakar, S. (2020) · 2020
Later among the works it cites.
Annotator Rationales for Labeling Tasks in Crowdsourcing
Kutlu, M., McDonnell, T., Elsayed, T., and Lease, M. (2020) · 2020
Later among the works it cites.
Towards hybrid human-ai workflows for unknown unknown detection
Liu, A., Guerra, S., Fung, I., Matute, G., Kamar, E., and Lasecki, W. (2020) · 2020
Later among the works it cites.
Directions in abusive language training data, a systematic review: Garbage in, garbage out
Vidgen, B. and Derczynski, L. (2020) · 2020
Later among the works it cites.
Discovering and validating ai errors with crowdsourced failure reports
Cabrera, A. A., Druck, A. J., Hong, J. I., and Perer, A. (2021) · 2021
Closest in time.
A checklist to combat cognitive biases in crowdsourcing
Draws, T., Rieger, A., Inel, O., Gadiraju, U., and Tintarev, N. (2021) · 2021
Closest in time.
Diversity in sociotechnical machine learning systems
[Dataset] Fazelpour, S. and De-Arteaga, M. (2021) · 2021
Closest in time.
Task design for crowdsourcing complex cognitive skills
Huang, G., Wu, M.-H., and Quinn, A. J. (2021) · 2021
Closest in time.