Fetching the paper…
Reading the bibliography…
In NLP annotation, it is common to have multiple annotators label the text and then obtain the ground truth labels based on the agreement of major annotators.
Roberta: A robustly optimized bert pretraining approach
Liu, Y.; Ott, M.; Goyal, N.; Du, J.; Joshi, M.; Chen, D.; Levy, O.; Lewis, M.; Zettlemoyer, L.; and Stoyanov, V. 2019 · 1907
Earlier work this paper cites.
Albert: A lite bert for self-supervised learning of language representations
Lan, Z.; Chen, M.; Goodman, S.; Gimpel, K.; Sharma, P.; and Soricut, R. 2019 · 1909
Earlier work this paper cites.
Social Bias Frames: Reasoning about Social and Power Implications of Language
Sap, M.; Gabriel, S.; Qin, L.; Jurafsky, D.; Smith, N. A.; and Choi, Y. 2020 · 1911
Earlier work this paper cites.
A coefficient of agreement for nominal scales
Cohen, J. 1960 · 1960
Earlier work this paper cites.
Measuring nominal scale agreement among many raters
Fleiss, J. L. 1971 · 1971
Earlier work this paper cites.
Mapping the margins: Intersectionality, identity politics, and violence against women of color
Crenshaw, K. 1990 · 1990
Earlier work this paper cites.
Aligning AI With Shared Human Values
Hendrycks, D.; Burns, C.; Basart, S.; Critch, A.; Li, J. Z.; Song, D. X.; and Steinhardt, J. 2021 · 2008
Earlier work this paper cites.
Exploiting ‘Subjective’ Annotations
Reidsma, D.; and op den Akker, R. 2008 · 2008
Earlier work this paper cites.
Subjective Natural Language Problems: Motivations, Applications, Characterizations, and Implications
Alm, C. O. 2011 · 2011
Earlier work this paper cites.
Social Chemistry 101: Learning to Reason about Social and Moral Norms
Forbes, M.; Hwang, J. D.; Shwartz, V.; Sap, M.; and Choi, Y. 2020 · 2011
Cited alongside, same era.
DynaSent: A Dynamic Benchmark for Sentiment Analysis
Potts, C.; Wu, Z.; Geiger, A.; and Kiela, D. 2021 · 2012
Cited alongside, same era.
A computational approach to politeness with application to social factors
Danescu-Niculescu-Mizil, C.; Sudhof, M.; Jurafsky, D.; Leskovec, J.; and Potts, C. 2013 · 2013
Cited alongside, same era.
Adam: A method for stochastic optimization
Kingma, D. P.; and Ba, J. 2015 · 2015
Cited alongside, same era.
”Why Should I Trust You?”: Explaining the Predictions of Any Classifier
Ribeiro, M. T.; Singh, S.; and Guestrin, C. 2016 · 2016
Cited alongside, same era.
Interrater disagreement resolution: A systematic procedure to reach consensus in annotation tasks
Oortwijn, Y.; Ossenkoppele, T.; and Betti, A. 2021 · 2021
Later among the works it cites.
On Releasing Annotator-Level Labels and Information in Datasets
Prabhakaran, V.; Davani, A. M.; and D’iaz, M. 2021 · 2021
Later among the works it cites.
SemEval-2021 Task 12: Learning with Disagreements
Uma, A.; Fornaciari, T.; Dumitrache, A.; Miller, T.; Chamberlain, J. P.; Plank, B.; Simpson, E.; and Poesio, M. 2021 · 2021
Later among the works it cites.
Dealing with disagreements: Looking beyond the majority vote in subjective annotations
Davani, A. M.; Díaz, M.; and Prabhakaran, V. 2022 · 2022
Later among the works it cites.
Jury Learning: Integrating Dissenting Voices into Machine Learning Models
Gordon, M. L.; Lam, M. S.; Park, J. S.; Patel, K.; Hancock, J. T.; Hashimoto, T.; and Bernstein, M. S. 2022 · 2022
Later among the works it cites.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2018 · 2018
Cited alongside, same era.
Explainable Agreement through Simulation for Tasks with Subjective Labels
Foley, J. 2018 · 2018
Cited alongside, same era.
Xlnet: Generalized autoregressive pretraining for language understanding
Yang, Z.; Dai, Z.; Yang, Y.; Carbonell, J.; Salakhutdinov, R. R.; and Le, Q. V. 2019 · 2019
Cited alongside, same era.
Scruples: A Corpus of Community Ethical Judgments on 32, 000 Real-Life Anecdotes
Lourie, N.; Bras, R. L.; and Choi, Y. 2021 · 2021
Cited alongside, same era.
Two Contrasting Data Annotation Paradigms for Subjective NLP Tasks
Röttger, P.; Vidgen, B.; Hovy, D.; and Pierrehumbert, J. B. 2022 · 2022
Later among the works it cites.
Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection
Sap, M.; Swayamdipta, S.; Vianna, L.; Zhou, X.; Choi, Y.; and Smith, N. A. 2022 · 2022
Later among the works it cites.
Scaling and Disagreements: Bias, Noise, and Ambiguity
Uma, A.; Almanea, D.; and Poesio, M. 2022 · 2022
Later among the works it cites.
Hate Speech and Counter Speech Detection: Conversational Context Does Matter
Yu, X. 2022 · 2022
Later among the works it cites.