Fetching the paper…
Reading the bibliography…
Annotator disagreement is ubiquitous in natural language processing (NLP) tasks.
Roberta: A robustly optimized bert pretraining approach
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019 · 1907
Earlier work this paper cites.
Maximum likelihood estimation of observer error-rates using the em algorithm
Alexander Philip Dawid and Allan M Skene. 1979 · 1979
Earlier work this paper cites.
Least squares quantization in pcm
Stuart Lloyd. 1982 · 1982
Earlier work this paper cites.
Personality similarity in twins reared apart and together
Auke Tellegen, David T Lykken, Thomas J Bouchard, Kimerly J Wilcox, Nancy L Segal, and Stephen Rich. 1988 · 1988
Earlier work this paper cites.
Emotion and adaptation
Richard S Lazarus. 1991 · 1991
Earlier work this paper cites.
Genetics, temperament, and personality
David C Rowe. 1997 · 1997
Earlier work this paper cites.
Happy families: A twin study of humour
Lynn Cherkas, Fran Hochberg, Alex J MacGregor, Harold Snieder, and Tim D Spector. 2000 · 2000
Earlier work this paper cites.
Intergroup bias
Miles Hewstone, Mark Rubin, and Hazel Willis. 2002 · 2002
Earlier work this paper cites.
The social identity theory of intergroup behavior
Henri Tajfel and John C Turner. 2004 · 2004
Earlier work this paper cites.
Interrater reliability: the kappa statistic
Mary L McHugh. 2012 · 2012
Earlier work this paper cites.
Multiplicity and word sense: evaluating and learning from multiply labeled word sense annotations
Rebecca J Passonneau, Vikas Bhardwaj, Ansaf Salleb-Aouissi, and Nancy Ide. 2012 · 2012
Earlier work this paper cites.
Corpus annotation through crowdsourcing: Towards best practice guidelines
Marta Sabou, Kalina Bontcheva, Leon Derczynski, and Arno Scharl. 2014 · 2014
Earlier work this paper cites.
Personality, humor styles and happiness: Happy people have positive humor styles
Thomas E Ford, Shaun K Lappi, and Christopher J Holden. 2016 · 2016
Earlier work this paper cites.
Social identity theory
Michael A Hogg. 2016 · 2016
Earlier work this paper cites.
Most “babies” are “little” and most “problems” are “huge”: Compositional entailment in adjective-nouns
Ellie Pavlick and Chris Callison-Burch. 2016 · 2016
Earlier work this paper cites.
Are you a racist or am I seeing things? annotator influence on hate speech detection on Twitter
Zeerak Waseem. 2016 · 2016
Earlier work this paper cites.
The self-concept: Social product and social force
Morris Rosenberg. 2017 · 2017
Earlier work this paper cites.
Addressing age-related bias in sentiment analysis
Mark Díaz, Isaac Johnson, Amanda Lazar, Anne Marie Piper, and Darren Gergle. 2018 · 2018
Earlier work this paper cites.
The psychology of humor: An integrative approach
Rod A Martin and Thomas Ford. 2018 · 2018
Cited alongside, same era.
The commitmentbank: Investigating projection in naturally occurring discourse
Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. 2019 · 2019
Cited alongside, same era.
BERT: Pre-training of deep bidirectional transformers for language understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 · 2019
Cited alongside, same era.
Are we modeling the task or the annotator? an investigation of annotator bias in natural language understanding datasets
Mor Geva, Yoav Goldberg, and Jonathan Berant. 2019 · 2019
Cited alongside, same era.
Evaluating BERT for natural language inference: A case study on the CommitmentBank
Nanjiang Jiang and Marie-Catherine de Marneffe. 2019b · 2019
Cited alongside, same era.
Learning personal human biases and representations for subjective tasks in natural language processing
Jan Kocoń, Marcin Gruza, Julita Bielaniewicz, Damian Grimling, Kamil Kanclerz, Piotr Miłkowski, and Przemysław Kazienko. 2021 · 2021
Later among the works it cites.
Designing toxic content classification for a diversity of perspectives
Deepak Kumar, Patrick Gage Kelley, Sunny Consolvo, Joshua Mason, Elie Bursztein, Zakir Durumeric, Kurt Thomas, and Michael Bailey. 2021 · 2021
Later among the works it cites.
Agreeing to disagree: Annotating offensive language datasets with annotators’ disagreement
Elisa Leonardelli, Stefano Menini, Alessio Palmero Aprosio, Marco Guerini, and Sara Tonelli. 2021 · 2021
Later among the works it cites.
Embracing ambiguity: Shifting the training target of NLI models
Johannes Mario Meissner, Napat Thumwanit, Saku Sugawara, and Akiko Aizawa. 2021 · 2021
Later among the works it cites.
Aggregating and learning from multiple annotators
Silviu Paun and Edwin Simpson. 2021 · 2021
Later among the works it cites.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Tonglin Jiang, Hao Li, and Yubo Hou. 2019 · 2019
Cited alongside, same era.
Inherent disagreements in human textual inferences
Ellie Pavlick and Tom Kwiatkowski. 2019 · 2019
Cited alongside, same era.
Predicting humorousness and metaphor novelty with Gaussian process preference learning
Edwin Simpson, Erik-Lân Do Dinh, Tristan Miller, and Iryna Gurevych. 2019 · 2019
Cited alongside, same era.
Identifying and measuring annotator bias based on annotators’ demographic characteristics
Hala Al Kuwatly, Maximilian Wich, and Georg Groh. 2020 · 2020
Cited alongside, same era.
GoEmotions: A dataset of fine-grained emotions
Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. 2020 · 2020
Cited alongside, same era.
AmbigQA: Answering ambiguous open-domain questions
Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020 · 2020
Cited alongside, same era.
What can we learn from collective human opinions on natural language inference data?
Yixin Nie, Xiang Zhou, and Mohit Bansal. 2020 · 2020
Cited alongside, same era.
Learning from disagreement: A survey
Alexandra N Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. 2021 · 2021
Later among the works it cites.
Dcn v2: Improved deep & cross network and practical lessons for web-scale learning to rank systems
Ruoxi Wang, Rakesh Shivanna, Derek Cheng, Sagar Jain, Dong Lin, Lichan Hong, and Ed Chi. 2021 · 2021
Later among the works it cites.
Learning with different amounts of annotation: From zero to many labels
Shujian Zhang, Chengyue Gong, and Eunsol Choi. 2021 · 2021
Later among the works it cites.
Identifying inherent disagreement in natural language inference
Xinliang Frederick Zhang and Marie-Catherine de Marneffe. 2021 · 2021
Later among the works it cites.
Analyzing the effects of annotator gender across nlp tasks
Laura Biester, Vanita Sharma, Ashkan Kazemi, Naihao Deng, Steven Wilson, and Rada Mihalcea. 2022 · 2022
Later among the works it cites.
Dealing with disagreements: Looking beyond the majority vote in subjective annotations
Aida Mostafazadeh Davani, Mark Díaz, and Vinodkumar Prabhakaran. 2022 · 2022
Later among the works it cites.
Jury learning: Integrating dissenting voices into machine learning models
Mitchell L Gordon, Michelle S Lam, Joon Sung Park, Kayur Patel, Jeff Hancock, Tatsunori Hashimoto, and Michael S Bernstein. 2022 · 2022
Later among the works it cites.
Investigating reasons for disagreement in natural language inference
Nan-Jiang Jiang and Marie-Catherine de Marneffe. 2022 · 2022
Later among the works it cites.
Handling and presenting harmful text in NLP research
Hannah Kirk, Abeba Birhane, Bertie Vidgen, and Leon Derczynski. 2022 · 2022
Later among the works it cites.
The “problem” of human label variation: On ground truth in data, modeling and evaluation
Barbara Plank. 2022 · 2022
Later among the works it cites.
Unifying data perspectivism and personalization: An application to social norms
Joan Plepi, Béla Neuendorf, Lucie Flek, and Charles Welch. 2022 · 2022
Later among the works it cites.
Annotators with attitudes: How annotator beliefs and identities bias toxic language detection
Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A. Smith. 2022 · 2022
Later among the works it cites.
Distributed NLI: Learning to predict human opinion distributions for language reasoning
Xiang Zhou, Yixin Nie, and Mohit Bansal. 2022 · 2022
Later among the works it cites.