Fetching the paper…
Reading the bibliography…
When groups of people are tasked with making a judgment, the issue of uncertainty often arises.
Discovery and presentation of evidence in adversary and nonadversary proceedings
E Allan Lind, John Thibaut, and Laurens Walker. 1973 · 1973
Earlier work this paper cites.
Judgment under Uncertainty: Heuristics and Biases
Amos Tversky and Daniel Kahneman. 1974 · 1974
Earlier work this paper cites.
Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm
A. Philip Dawid and Allan Skene. 1979 · 1979
Earlier work this paper cites.
A strictly evolutionary model of common law
R Peter Terrebonne. 1981 · 1981
Earlier work this paper cites.
Notetaking can aid juror recall
David L. Rosenhan, Sara L. Eisner, and Robert J. Robinson. 1994 · 1994
Earlier work this paper cites.
WordNet: A Lexical Database for English
George A. Miller. 1995 · 1995
Earlier work this paper cites.
Aleatory and epistemic uncertainty in probability elicitation with an example from hazardous waste management
Stephen C. Hora. 1996 · 1996
Earlier work this paper cites.
Development and Use of a Gold-Standard Data Set for Subjectivity Classifications. In Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics . Association for Computational Linguistics, College Park, Maryland, USA, 246–253
Janyce M. Wiebe, Rebecca F. Bruce, and Thomas P. O’Hara. 1999 · 1999
Earlier work this paper cites.
Placing Search in Context: The Concept Revisited
Lev Finkelstein, Evgeniy Gabrilovich, Y. Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. 2002 · 2002
Earlier work this paper cites.
Defining Uncertainty: A Conceptual Basis for Uncertainty Management in Model-Based Decision Support
Warren E. Walker, Poul Harremoës, Jan Rotmans, Jeroen P. van der Sluijs, M. B. A. Asselt, Paul Janssen, and Martin Krayer von Krauss. 2003 · 2003
Earlier work this paper cites.
Ranking with Uncertain Labels
Shuicheng Yan, Huan Wang, Thomas S. Huang, Qiong Yang, and Xiaoou Tang. 2007 · 2007
Earlier work this paper cites.
Cheap and Fast – But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing . Association for Computational Linguistics, Honolulu, Hawaii, 254–263
Rion Snow, Brendan O’Connor, Daniel Jurafsky, and Andrew Ng. 2008 · 2008
Earlier work this paper cites.
ImageNet: A large-scale hierarchical image database
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, K. Li, and Li Fei-Fei. 2009 · 2009
Earlier work this paper cites.
Aleatory or epistemic? Does it matter?
Armen Der Kiureghian and O. Ditlevsen. 2009 · 2009
Earlier work this paper cites.
Quality Management on Amazon Mechanical Turk. In Proceedings of the ACM SIGKDD Workshop on Human Computation (HCOMP ’10) . Association for Computing Machinery, New York, NY, USA, 64–67
Panagiotis G. Ipeirotis, Foster Provost, and Jing Wang. 2010 · 2010
Earlier work this paper cites.
Crowds in Two Seconds: Enabling Realtime Crowd-Powered Interfaces. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (UIST ’11) . Association for Computing Machinery, New York, NY, USA, 33–42
Michael S. Bernstein, Joel Brandt, Robert C. Miller, and David R. Karger. 2011 · 2011
Earlier work this paper cites.
Distinguishing two dimensions of uncertainty
Craig R Fox and Gülden Ülkümen. 2011 · 2011
Earlier work this paper cites.
ConsiderIt: Improving Structured Public Deliberation. In CHI ’11 Extended Abstracts on Human Factors in Computing Systems (CHI EA ’11) . ACM, New York, NY, USA, 1831–1836
Travis Kriplean, Jonathan T. Morgan, Deen Freelon, Alan Borning, and Lance Bennett. 2011 · 2011
Earlier work this paper cites.
How much spam can you take? an analysis of crowdsourcing results to increase accuracy. In Proc. ACM SIGIR Workshop on Crowdsourcing for Information Retrieval (CIR’11) . 21–26
Jeroen Vuurens, Arjen P de Vries, and Carsten Eickhoff. 2011 · 2011
Earlier work this paper cites.
Mechanical cheat: Spamming schemes and adversarial techniques on crowdsourcing platforms. In CrowdSearch
Djellel Eddine Difallah, Gianluca Demartini, and Philippe Cudré-Mauroux. 2012 · 2012
Earlier work this paper cites.
Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial
Kevin A. Hallgren. 2012 · 2012
Earlier work this paper cites.
Toward a synthesis of cognitive biases: how noisy information processing can bias human decision making
Martin Hilbert. 2012 · 2012
Earlier work this paper cites.
Improving Word Representations via Global Context and Multiple Word Prototypes. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . Association for Computational Linguistics, Jeju Island, Korea, 873–882
Eric Huang, Richard Socher, Christopher Manning, and Andrew Ng. 2012 · 2012
Earlier work this paper cites.
Crowd truth: Harnessing disagreement in crowdsourcing a relation extraction gold standard
Lora Aroyo and Chris Welty. 2013 · 2013
Earlier work this paper cites.
Online deliberation design: Choices, criteria, and evidence
Todd Davies and Reid Chandler. 2013 · 2013
Earlier work this paper cites.
The Benefits of a Model of Annotation
R. Passonneau and Bob Carpenter. 2013 · 2013
Earlier work this paper cites.
Evaluating consent and legitimacy amongst shifting community norms: an EVE Online case study
Nicolas Suzor and Darryl Woodford. 2013 · 2013
Earlier work this paper cites.
Sockpuppet Detection in Wikipedia: A Corpus of Real-World Deceptive Writing for Linking Identities. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14) . European Language Resources Association (ELRA), Reykjavik, Iceland, 1355–1358
Thamar Solorio, Ragib Hasan, and Mainul Mizan. 2014 · 2014
Earlier work this paper cites.
Training workers for improving performance in crowdsourcing microtasks. In European Conference on Technology Enhanced Learning . Springer, 100–114
Ujwal Gadiraju, Besnik Fetahu, and Ricardo Kawase. 2015 · 2015
Earlier work this paper cites.
Crowdsourcing from Scratch: A Pragmatic Experiment in Data Collection by Novice Requesters. In HCOMP
Alexandra Papoutsaki, Hua Guo, Danaë Metaxa-Kakavouli, Connor Gramazio, Jeff Rasley, Wenting Xie, Guan Wang, and Jeff Huang. 2015 · 2015
Earlier work this paper cites.
ImageNet Large Scale Visual Recognition Challenge
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li Fei-Fei. 2015 · 2015
Earlier work this paper cites.
Microtalk: Using argumentation to improve crowdsourcing accuracy. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing , Vol. 4. 32–41
Ryan Drapeau, Lydia Chilton, Jonathan Bragg, and Daniel Weld. 2016 · 2016
Earlier work this paper cites.
Understanding Crowdsourcing Workflow: Modeling and Optimizing Iterative and Parallel Processes. In AAAI Conference on Human Computation & Crowdsourcing
Shinsuke Goto, Toru Ishida, and Donghui Lin. 2016 · 2016
Cited alongside, same era.
Here’s Why Facebook Removing That Vietnam War Photo Is So Important
Matthew Ingram. [n. d.] · 2016
Cited alongside, same era.
Parting crowds: Characterizing divergent interpretations in crowdsourced annotation tasks. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing . 1637–1648
Sanjay Kairam and Jeffrey Heer. 2016 · 2016
Cited alongside, same era.
Effective Crowd Annotation for Relation Extraction. In Proceedings of NAACL and HLT 2016
Angli Liu, Stephen Soderland, Jonathan Bragg, Christopher H. Lin, Xiao Ling, and Daniel S. Weld. 2016 · 2016
Cited alongside, same era.
After outcry, Facebook will reinstate iconic Vietnam War photo
Jethro Mullen and Charles Riley. [n. d.] · 2016
The challenges of responding to misinformation during a pandemic: content moderation and the limitations of the concept of harm
Stephanie Alice Baker, Matthew Wade, and Michael James Walsh. 2020 · 2020
Later among the works it cites.
Lucas Beyer, Olivier J. H’enaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. 2020 · 2020
Later among the works it cites.
Introduction to Statistics in Metrology
Stephen Crowder, Collin Delker, Eric Forrest, and Nevin Martin. 2020 · 2020
Later among the works it cites.
Digital Juries: A Civics-Oriented Approach to Platform Governance
Jenny Fan and Amy X. Zhang. 2020 · 2020
Later among the works it cites.
Garbage in, garbage out?: do machine learning application papers in social computing report where human-labeled training data comes from?
R. Stuart Geiger, Kevin Yu, Yanlai Yang, Mindy Dai, Jie Qiu, Rebekah Tang, and Jenny Huang. 2020 · 2020
Later among the works it cites.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Cited alongside, same era.
Are You a Racist or Am I Seeing Things? Annotator Influence on Hate Speech Detection on Twitter. In Proceedings of the First Workshop on NLP and Computational Social Science . Association for Computational Linguistics, Austin, Texas, 138–142
Zeerak Waseem. 2016 · 2016
Cited alongside, same era.
Revolt: Collaborative crowdsourcing for labeling machine learning datasets. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems . 2334–2346
Joseph Chee Chang, Saleema Amershi, and Ece Kamar. 2017 · 2017
Cited alongside, same era.
Clarity is a Worthwhile Quality: On the Role of Task Clarity in Microtask Crowdsourcing. In Proceedings of the 28th ACM Conference on Hypertext and Social Media (HT ’17) . Association for Computing Machinery, New York, NY, USA, 5–14
Ujwal Gadiraju, Jie Yang, and Alessandro Bozzon. 2017 · 2017
Cited alongside, same era.
Harnessing Diversity in Crowds and Machines for Better NER Performance. In ESWC
Oana Inel and Lora Aroyo. 2017 · 2017
Cited alongside, same era.
Gradescope: A Fast, Flexible, and Fair System for Scalable Assessment of Handwritten Work. In Proceedings of the Fourth (2017) ACM Conference on Learning @ Scale (L@S ’17) . Association for Computing Machinery, New York, NY, USA, 81–88
Arjun Singh, Sergey Karayev, Kevin Gutowski, and Pieter Abbeel. 2017 · 2017
Cited alongside, same era.
Confusing the Crowd: Task Instruction Quality on Amazon Mechanical Turk. In HCOMP
Meng-Han Wu and Alexander J. Quinn. 2017 · 2017
Cited alongside, same era.
Position-aware Attention and Supervised Data Improve Slot Filling. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing . Association for Computational Linguistics, Copenhagen, Denmark, 35–45
Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. 2017b · 2017
Cited alongside, same era.
Controlling Bad Behavior in Online Communities: An Examination of Moderation Work. In International Conference on Interaction Sciences
Aiden R. McGillicuddy, Jean-Grégoire Bernard, and Jocelyn Cranefield. 2020 · 2020
Later among the works it cites.
Toxicity Detection: Does Context Really Matter?. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics . Association for Computational Linguistics, Online, 4296–4305
John Pavlopoulos, Jeffrey Sorensen, Lucas Dixon, Nithum Thain, and Ion Androutsopoulos. 2020 · 2020
Later among the works it cites.
Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) . Association for Computational Linguistics, Online, 9275–9293
Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith, and Yejin Choi. 2020 · 2020
Later among the works it cites.
Misinformation encountered during a simulated jury deliberation can distort jurors’ memory of a trial and bias their verdicts
Craig Thorley, Lara Beaton, Phillip Deguara, Brittany Jerome, Dua Khan, and Kaela Schopp. 2020 · 2020
Later among the works it cites.
Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty
Umang Bhatt, Javier Antorán, Yunfeng Zhang, Q. Vera Liao, Prasanna Sattigeri, Riccardo Fogliato, Gabrielle Melançon, Ranganath Krishnan, Jason Stanley, Omesh Tickoo, Lama Nachman, Rumi Chunara, Madhulika Srikumar, Adrian Weller, and Alice Xiang. 2021 · 2021
Later among the works it cites.
The Impossibility of Automating Ambiguity
Abeba Birhane. 2021 · 2021
Later among the works it cites.
Variations in guidelines for diagnosis of child physical abuse in high-income countries: a systematic review
Flora Blangis, Slimane Allali, Jérémie F Cohen, Nathalie Vabres, Catherine Adamsbaum, Caroline Rey-Salmon, Andreas Werner, Yacine Refes, Pauline Adnot, Christèle Gras-Le Guen, et al · 2021
Later among the works it cites.
My Team Will Go On: Differentiating High and Low Viability Teams through Team Interaction
Hancheng Cao, Vivian Yang, Victor Chen, Yu Jin Lee, Lydia Stone, N’godjigui Junior Diarrassouba, Mark E. Whiting, and Michael S. Bernstein. 2021 · 2021
Later among the works it cites.
Goldilocks: Consistent Crowdsourced Scalar Annotations with Relative Uncertainty
Quan Ze Chen, Daniel S. Weld, and Amy X. Zhang. 2021 · 2021
Later among the works it cites.
Inconsistency in Conference Peer Review: Revisiting the 2014 NeurIPS Experiment
Corinna Cortes and Neil D. Lawrence. 2021 · 2021
Later among the works it cites.
Beyond Black & White: Leveraging Annotator Disagreement via Soft-Label Multi-Task Learning. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies . Association for Computational Linguistics, Online, 2591–2597
Tommaso Fornaciari, Alexandra Uma, Silviu Paun, Barbara Plank, Dirk Hovy, and Massimo Poesio. 2021 · 2021
Later among the works it cites.
Datasheets for datasets
Timnit Gebru, Jamie H. Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna M. Wallach, Hal Daumé, and Kate Crawford. 2021 · 2021
Later among the works it cites.
The Disagreement Deconvolution: Bringing Machine Learning Performance Metrics In Line With Reality. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (CHI ’21) . Association for Computing Machinery, New York, NY, USA, Article 388, 14 pages
Mitchell L. Gordon, Kaitlyn Zhou, Kayur Patel, Tatsunori Hashimoto, and Michael S. Bernstein. 2021 · 2021
Later among the works it cites.
The Challenge of Variable Effort Crowdsourcing and How Visible Gold Can Help
Danula Hettiachchi, Mike Schaekermann, Tristan J. McKinney, and Matthew Lease. 2021 · 2021
Later among the works it cites.
Understanding international perceptions of the severity of harmful content online
Jialun Aaron Jiang, Morgan Klaus Scheuerman, Casey Fiesler, and Jed R. Brubaker. 2021 · 2021
Later among the works it cites.
Agreeing to Disagree: Annotating Offensive Language Datasets with Annotators’ Disagreement. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing . Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 10528–10539
Elisa Leonardelli, Stefano Menini, Alessio Palmero Aprosio, Marco Guerini, and Sara Tonelli. 2021 · 2021
Later among the works it cites.
Stefano Menini, Alessio Palmero Aprosio, and Sara Tonelli. 2021 · 2021
Later among the works it cites.
On Releasing Annotator-Level Labels and Information in Datasets. In Proceedings of The Joint 15th Linguistic Annotation Workshop (LAW) and 3rd Designing Meaning Representations (DMR) Workshop . Association for Computational Linguistics, Punta Cana, Dominican Republic, 133–138
Vinodkumar Prabhakaran, Aida Mostafazadeh Davani, and Mark Diaz. 2021 · 2021
Later among the works it cites.
Vivek Pradhan, Mike Schaekermann, and Matthew Lease. 2021 · 2021
Later among the works it cites.
Annotators with attitudes: How annotator beliefs and identities bias toxic language detection
Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A Smith. 2021 · 2021
Later among the works it cites.
Re-TACRED: Addressing Shortcomings of the TACRED Dataset. In AAAI Conference on Artificial Intelligence
George Stoica, Emmanouil Antonios Platanios, and Barnab’as P’oczos. 2021 · 2021
Later among the works it cites.
Deep neural learning on weighted datasets utilizing label disagreement from crowdsourcing
Dongsheng Wang, Prayag Tiwari, Mohammad Shorfuzzaman, and Ingo Schmitt. 2021 · 2021
Later among the works it cites.
What is the Will of the People? Moderation Preferences for Misinformation
Shubham Atreja, Libby Hemphill, and Paul Resnick. 2022 · 2022
Later among the works it cites.
Eliciting and Learning with Soft Labels from Every Annotator. In HCOMP
Katherine M. Collins, Umang Bhatt, and Adrian Weller. 2022 · 2022
Later among the works it cites.
Annotation Curricula to Implicitly Train Non-Expert Annotators
Ji-Ung Lee, Jan-Christoph Klie, and Iryna Gurevych. 2022 · 2022
Later among the works it cites.
Modes of Uncertainty in HCI
Robert Soden, Laura Devendorf, Richmond Y. Wong, Yoko Akama, and Ann Light. 2022 · 2022
Later among the works it cites.