Fetching the paper…
Reading the bibliography…
In subjective NLP tasks, where a single ground truth does not exist, the inclusion of diverse annotators becomes crucial as their unique perspectives significantly influence the annotations.
Roberta: A robustly optimized bert pretraining approach
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019 · 1907
Earlier work this paper cites.
Measuring nominal scale agreement among many raters
Joseph L Fleiss. 1971 · 1971
Earlier work this paper cites.
Moral foundations theory: The pragmatic validity of moral pluralism
Jesse Graham, Jonathan Haidt, Sena Koleva, Matt Motyl, Ravi Iyer, Sean P Wojcik, and Peter H Ditto. 2013 · 2013
Earlier work this paper cites.
Linguistically debatable or just plain wrong?
Barbara Plank, Dirk Hovy, and Anders Søgaard. 2014 · 2014
Earlier work this paper cites.
Cultural differences in moral judgment and behavior, across and within societies
Jesse Graham, Peter Meindl, Erica Beall, Kate M Johnson, and Li Zhang. 2016 · 2016
Earlier work this paper cites.
Hateful symbols or hateful people? predictive features for hate speech detection on twitter
Zeerak Talat and Dirk Hovy. 2016 · 2016
Earlier work this paper cites.
Short and extra-short forms of the big five inventory–2: The bfi-2-s and bfi-2-xs
Christopher J Soto and Oliver P John. 2017 · 2017
Earlier work this paper cites.
Gender shades: Intersectional accuracy disparities in commercial gender classification
Joy Buolamwini and Timnit Gebru. 2018 · 2018
Earlier work this paper cites.
Addressing age-related bias in sentiment analysis
Mark Díaz, Isaac Johnson, Amanda Lazar, Anne Marie Piper, and Darren Gergle. 2018 · 2018
Earlier work this paper cites.
The gab hate corpus: A collection of 27k posts annotated for hate speech
Brendan Kennedy, Mohammad Atari, Aida Mostafazadeh Davani, Leigh Yeh, Ali Omrani, Yehsong Kim, Kris Coombs, Shreya Havaldar, Gwenyth Portillo-Wightman, Elaine Gonzalez, et al. 2018 · 2018
Earlier work this paper cites.
A new measure of polarization in the annotation of hate speech
Sohail Akhtar, Valerio Basile, and Viviana Patti. 2019 · 2019
Earlier work this paper cites.
Incorporating demographic embeddings into language understanding
Justin Garten, Brendan Kennedy, Joe Hoover, Kenji Sagae, and Morteza Dehghani. 2019 · 2019
Earlier work this paper cites.
Inherent disagreements in human textual inferences
Ellie Pavlick and Tom Kwiatkowski. 2019 · 2019
Earlier work this paper cites.
Human uncertainty makes classification more robust
Joshua C Peterson, Ruairidh M Battleday, Thomas L Griffiths, and Olga Russakovsky. 2019 · 2019
Cited alongside, same era.
Modeling annotator perspective and polarized opinions to improve hate speech detection
Sohail Akhtar, Valerio Basile, and Viviana Patti. 2020 · 2020
Cited alongside, same era.
It’s the end of the gold standard as we know it. on the impact of pre-aggregation on the evaluation of highly subjective tasks
Valerio Basile. 2020 · 2020
Cited alongside, same era.
Sohail Akhtar, Valerio Basile, and Viviana Patti. 2021 · 2021
Cited alongside, same era.
Toward a perspectivist turn in ground truthing for predictive computing
Valerio Basile, Federico Cabitza, Andrea Campagner, and Michael Fell. 2021 · 2021
The “problem” of human label variation: On ground truth in data, modeling and evaluation
Barbara Plank. 2022 · 2022
Later among the works it cites.
The origin and value of disagreement among data labelers: A case study of individual differences in hate speech annotation
Yisi Sang and Jeffrey Stanton. 2022 · 2022
Later among the works it cites.
The moral foundations reddit corpus
Jackson Trager, Alireza S Ziabari, Aida Mostafazadeh Davani, Preni Golazazian, Farzan Karimi-Malekabadi, Ali Omrani, Zhihe Li, Brendan Kennedy, Nils Karl Reimer, Melissa Reyes, et al. 2022 · 2022
Later among the works it cites.
Actor: Active learning with annotator-specific classification heads to embrace human label variation
Xinpeng Wang and Barbara Plank. 2023 · 2022
Later among the works it cites.
Morality beyond the weird: How the nomological network of morality varies across cultures
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Cited alongside, same era.
Did they answer? subjective acts and intents in conversational discourse
Elisa Ferracane, Greg Durrett, Junyi Jessy Li, and Katrin Erk. 2021 · 2021
Cited alongside, same era.
Lora: Low-rank adaptation of large language models
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. 2021 · 2021
Cited alongside, same era.
Reconsidering annotator disagreement about racist language: Noise or signal?
Savannah Larimore, Ian Kennedy, Breon Haskett, and Alina Arseniev-Koehler. 2021 · 2021
Cited alongside, same era.
On releasing annotator-level labels and information in datasets
Vinodkumar Prabhakaran, Aida Mostafazadeh Davani, and Mark Diaz. 2021 · 2021
Cited alongside, same era.
Annotators with attitudes: How annotator beliefs and identities bias toxic language detection
Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A Smith. 2021 · 2021
Cited alongside, same era.
Learning from disagreement: A survey
Alexandra N Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. 2021 · 2021
Cited alongside, same era.
Dealing with disagreements: Looking beyond the majority vote in subjective annotations
Aida Mostafazadeh Davani, Mark Díaz, and Vinodkumar Prabhakaran. 2022 · 2022
Cited alongside, same era.
Mohammad Atari, Jonathan Haidt, Jesse Graham, Sena Koleva, Sean T Stevens, and Morteza Dehghani. 2023 · 2023
Later among the works it cites.
Which examples should be multiply annotated? active learning when annotators may disagree
Connor Baumler, Anna Sotnikova, and Hal Daumé III. 2023 · 2023
Later among the works it cites.
Confidence-based ensembling of perspective-aware models
Silvia Casola, Soda Lo, Valerio Basile, Simona Frenda, Alessandra Cignarella, Viviana Patti, and Cristina Bosco. 2023 · 2023
Later among the works it cites.
Hate speech classifiers learn normative social stereotypes
Aida Mostafazadeh Davani, Mohammad Atari, Brendan Kennedy, and Morteza Dehghani. 2023 · 2023
Later among the works it cites.
You are what you annotate: Towards better models through annotator representations
Naihao Deng, Xinliang Zhang, Siyang Liu, Winston Wu, Lu Wang, and Rada Mihalcea. 2023 · 2023
Later among the works it cites.
A survey of large language models
Zheng Liu et al. 2023 · 2023
Later among the works it cites.
A comprehensive overview of large language models
Humza Naveed, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, Saeed Anwar, Muhammad Usman, Nick Barnes, and Ajmal Mian. 2023 · 2023
Later among the works it cites.
Llama: Open and efficient foundation language models
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023 · 2023
Later among the works it cites.