Fetching the paper…
Reading the bibliography…
Label aggregation such as majority voting is commonly used to resolve annotator disagreement in dataset creation.
A framework for understanding unintended consequences of machine learning
Harini Suresh and John V Guttag. 2019 · 1901
Earlier work this paper cites.
Discriminative active learning
Daniel Gissin and Shai Shalev-Shwartz. 2019 · 1907
Earlier work this paper cites.
Committee-based sampling for training probabilistic classifiers
Ido Dagan and Sean P Engelson. 1995 · 1995
Earlier work this paper cites.
A sequential algorithm for training text classifiers: Corrigendum and additional data
David D Lewis. 1995 · 1995
Earlier work this paper cites.
Analyzing disagreements
Beata Beigman Klebanov, Eyal Beigman, and Daniel Diermeier. 2008 · 2008
Earlier work this paper cites.
Squibs: Reliability measurement without limits
Dennis Reidsma and Jean Carletta. 2008 · 2008
Earlier work this paper cites.
Get another label? improving data quality and data mining using multiple, noisy labelers
Victor S. Sheng, Foster Provost, and Panagiotis G. Ipeirotis. 2008 · 2008
Earlier work this paper cites.
Bayesian active learning for classification and preference learning
Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Lengyel. 2011 · 2011
Earlier work this paper cites.
Subjective natural language problems: Motivations, applications, characterizations, and implications
Cecilia Ovesdotter Alm. 2011 · 2011
Earlier work this paper cites.
Modelling annotator bias with multi-task Gaussian processes: An application to machine translation quality estimation
Trevor Cohn and Lucia Specia. 2013 · 2013
Earlier work this paper cites.
Learning whom to trust with MACE
Dirk Hovy, Taylor Berg-Kirkpatrick, Ashish Vaswani, and Eduard Hovy. 2013 · 2013
Earlier work this paper cites.
Linguistically debatable or just plain wrong?
Barbara Plank, Dirk Hovy, and Anders Søgaard. 2014 · 2014
Cited alongside, same era.
Truth is a lie: Crowd truth and the seven myths of human annotation
Lora Aroyo and Chris Welty. 2015 · 2015
Cited alongside, same era.
Dropout as a bayesian approximation: Representing model uncertainty in deep learning
Yarin Gal and Zoubin Ghahramani. 2016 · 2016
Cited alongside, same era.
A baseline for detecting misclassified and out-of-distribution examples in neural networks
Dan Hendrycks and Kevin Gimpel. 2017 · 2017
Cited alongside, same era.
Active learning for convolutional neural networks: A core-set approach
Ozan Sener and Silvio Savarese. 2017 · 2017
Cited alongside, same era.
Pushshift gab corpus
Gavin Gaffney Gaffney. 2018 · 2018
A case for soft loss functions
Alexandra Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. 2020 · 2020
Later among the works it cites.
Sohail Akhtar, Valerio Basile, and Viviana Patti. 2021 · 2021
Later among the works it cites.
Beyond black & white: Leveraging annotator disagreement via soft-label multi-task learning
Tommaso Fornaciari, Alexandra Uma, Silviu Paun, Barbara Plank, Dirk Hovy, and Massimo Poesio. 2021 · 2021
Later among the works it cites.
The disagreement deconvolution: Bringing machine learning performance metrics in line with reality
Mitchell L Gordon, Kaitlyn Zhou, Kayur Patel, Tatsunori Hashimoto, and Michael S Bernstein. 2021 · 2021
Later among the works it cites.
Learning from disagreement: A survey
Alexandra Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. 2021 · 2021
Later among the works it cites.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Cited alongside, same era.
Deep learning from crowds
Filipe Rodrigues and Francisco Pereira. 2018 · 2018
Cited alongside, same era.
BERT: Pre-training of deep bidirectional transformers for language understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 · 2019
Cited alongside, same era.
Inherent disagreements in human textual inferences
Ellie Pavlick and Tom Kwiatkowski. 2019 · 2019
Cited alongside, same era.
Human uncertainty makes classification more robust
Joshua C Peterson, Ruairidh M Battleday, Thomas L Griffiths, and Olga Russakovsky. 2019 · 2019
Cited alongside, same era.
Multi-annotator probabilistic active learning
Marek Herde, Daniel Kottke, Denis Huseljic, and Bernhard Sick. 2021 · 2020
Cited alongside, same era.
Disembodied machine learning: On the illusion of objectivity in nlp
Zeerak Waseem, Smarika Lulz, Joachim Bingel, and Isabelle Augenstein. 2021 · 2021
Later among the works it cites.
Cartography active learning
Mike Zhang and Barbara Plank. 2021 · 2021
Later among the works it cites.
Dealing with disagreements: Looking beyond the majority vote in subjective annotations
Aida Mostafazadeh Davani, Mark Díaz, and Vinodkumar Prabhakaran. 2022 · 2022
Later among the works it cites.
Introducing the gab hate corpus: defining and applying hate-based rhetoric to social media posts at scale
Brendan Kennedy, Mohammad Atari, Aida Mostafazadeh Davani, Leigh Yeh, Ali Omrani, Yehsong Kim, Kris Coombs, Shreya Havaldar, Gwenyth Portillo-Wightman, Elaine Gonzalez, et al. 2022 · 2022
Later among the works it cites.
Annotators with attitudes: How annotator beliefs and identities bias toxic language detection
Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A. Smith. 2022 · 2022
Later among the works it cites.
Which examples should be multiply annotated? active learning when annotators may disagree
Connor Baumler, Anna Sotnikova, and Hal Daumé III. 2023 · 2023
Closest in time.