Fetching the paper…
Reading the bibliography…
Human annotation plays a core role in machine learning -- annotations for supervised models, safety guardrails for generative models, and human feedback for reinforcement learning, to cite a few avenues.
Teoria statistica delle classi e calcolo delle probabilita
Carlo Bonferroni. 1936 · 1936
Earlier work this paper cites.
The negentropy principle of information
Leon Brillouin. 1953 · 1953
Earlier work this paper cites.
A simple sequentially rejective multiple test procedure
Sture Holm. 1979 · 1979
Earlier work this paper cites.
Controlling the false discovery rate: A practical and powerful approach to multiple testing
Yoav Benjamini and Yosef Hochberg. 1995 · 1995
Earlier work this paper cites.
Reliability in content analysis: Some common misconceptions and recommendations
Klaus Krippendorff. 2004 · 2004
Earlier work this paper cites.
The controversy over null hypothesis significance testing revisited
Nekane Balluerka, Juana Gómez, and Dolores Hidalgo. 2005 · 2005
Earlier work this paper cites.
Modularity and community structure in networks
Mark EJ Newman. 2006 · 2006
Earlier work this paper cites.
Toxicity detection: Does context really matter?
John Pavlopoulos, Jeffrey Sorensen, Lucas Dixon, Nithum Thain, and Ion Androutsopoulos. 2020 · 2006
Earlier work this paper cites.
Multiple testing for exploratory research
Jelle J. Goeman and Aldo Solari. 2011 · 2011
Earlier work this paper cites.
Detecting hate speech on the world wide web
William Warner and Julia Hirschberg. 2012 · 2012
Earlier work this paper cites.
Crowd truth: Harnessing disagreement in crowdsourcing a relation extraction gold standard
Lora Aroyo and Chris Welty. 2013 · 2013
Earlier work this paper cites.
Learning whom to trust with mace
Dirk Hovy, Taylor Berg-Kirkpatrick, Ashish Vaswani, and Eduard Hovy. 2013 · 2013
Earlier work this paper cites.
After statistics reform: Should we still teach significance testing?
Tony Hak. 2014 · 2014
Earlier work this paper cites.
Linguistically debatable or just plain wrong?
Barbara Plank, Dirk Hovy, and Anders Søgaard. 2014 · 2014
Earlier work this paper cites.
Truth is a lie: Crowd truth and the seven myths of human annotation
Lora Aroyo and Chris Welty. 2015 · 2015
Earlier work this paper cites.
Interpretation of statistical significance-exploratory versus confirmative testing in clinical trials, epidemiological studies, meta-analyses and toxicological screening (using Ginkgo biloba as an example)
Wilhelm Gaus, B Mayer, and R Muche. 2015 · 2015
Earlier work this paper cites.
Demographic factors improve classification performance
Dirk Hovy. 2015 · 2015
Earlier work this paper cites.
Editorial
David Trafimow and Michael Marks. 2015 · 2015
Earlier work this paper cites.
Parting crowds: Characterizing divergent interpretations in crowdsourced annotation tasks
Sanjay Kairam and Jeffrey Heer. 2016 · 2016
Earlier work this paper cites.
Are you a racist or am i seeing things? annotator influence on hate speech detection on twitter
Zeerak Waseem. 2016 · 2016
Earlier work this paper cites.
Hateful symbols or hateful people? predictive features for hate speech detection on twitter
Zeerak Waseem and Dirk Hovy. 2016 · 2016
Earlier work this paper cites.
Automated hate speech detection and the problem of offensive language
Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017 · 2017
Earlier work this paper cites.
Do p values lose their meaning in exploratory analyses? it depends how you define the familywise error rate
Mark Rubin. 2017 · 2017
Earlier work this paper cites.
Ex machina: Personal attacks seen at scale
Ellery Wulczyn, Nithum Thain, and Lucas Dixon. 2017 · 2017
Earlier work this paper cites.
Large scale crowdsourcing and characterization of twitter abusive behavior
Antigoni Founta, Constantinos Djouvas, Despoina Chatzakou, Ilias Leontiadis, Jeremy Blackburn, Gianluca Stringhini, Athena Vakali, Michael Sirivianos, and Nicolas Kourtellis. 2018 · 2018
Earlier work this paper cites.
Efficient elicitation approaches to estimate collective crowd answers
John Joon Young Chung, Jean Y Song, Sindhu Kutty, Sungsoo Hong, Juho Kim, and Walter S Lasecki. 2019 · 2019
Earlier work this paper cites.
Incorporating demographic embeddings into language understanding
Justin Garten, Brendan Kennedy, Joe Hoover, Kenji Sagae, and Morteza Dehghani. 2019 · 2019
Earlier work this paper cites.
Are we modeling the task or the annotator? an investigation of annotator bias in natural language understanding datasets
Mor Geva, Yoav Goldberg, and Jonathan Berant. 2019 · 2019
Earlier work this paper cites.
Learning to predict population-level label distributions
Tong Liu, Akash Venkatachalam, Pratik Sanjay Bongale, and Christopher M. Homan. 2019 · 2019
Cited alongside, same era.
Dissecting racial bias in an algorithm used to manage the health of populations
Ziad Obermeyer, Brian Powers, Christine Vogeli, and Sendhil Mullainathan. 2019 · 2019
Cited alongside, same era.
Online hate ratings vary by extremes: A statistical analysis
Joni Salminen, Hind Almerekhi, Ahmed Mohamed Kamel, Soon-gyo Jung, and Bernard J Jansen. 2019 · 2019
Cited alongside, same era.
Challenges and frontiers in abusive content detection
Bertie Vidgen, Alex Harris, Dong Nguyen, Rebekah Tromble, Scott Hale, and Helen Margetts. 2019 · 2019
Cited alongside, same era.
Identifying and measuring annotator bias based on annotators’ demographic characteristics
Hala Al Kuwatly, Maximilian Wich, and Georg Groh. 2020 · 2020
Cited alongside, same era.
Improving alignment of dialogue agents via targeted human judgements
Amelia Glaese, Nat McAleese, Maja Trębacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, Lucy Campbell-Gillingham, Jonathan Uesato, Po-Sen Huang, Ramona Comanescu, Fan Yang, Abigail See, Sumanth Dathathri, Rory Greig, Charlie Chen, Doug Fritz, Jaume Sanchez Elias, Richard Green, Soňa Mokrá, Nicholas Fernando, Boxi Wu, Rachel Foley, Susannah Young, Iason Gabriel, William Isaac, John Mellor, Demis Hassabis, Koray Kavukcuoglu, Lisa Anne Hendricks, and Geoffrey Irving. 2022 · 2022
Later among the works it cites.
Pathways language model (PaLM): Scaling to 540 billion parameters for breakthrough performance
Google. 2022 · 2022
Later among the works it cites.
Answer-level calibration for free-form multiple choice question answering
Sawan Kumar. 2022 · 2022
Later among the works it cites.
Introducing ChatGPT
OpenAI. 2022 · 2022
Later among the works it cites.
Training language models to follow instructions with human feedback
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
European Commission. 2020 · 2020
Cited alongside, same era.
Harmonization sometimes harms
Manfred Klenner, Anne Göhring, and Michael Amsler. 2020 · 2020
Cited alongside, same era.
Neighborhood-based pooling for population-level label distribution learning
Tharindu Cyril Weerasooriya, Tong Liu, and Christopher M. Homan. 2020 · 2020
Cited alongside, same era.
Investigating annotator bias with a graph-based approach
Maximilian Wich, Hala Al Kuwatly, and Georg Groh. 2020 · 2020
Cited alongside, same era.
Fine-tuning language models from human preferences
Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2020 · 2020
Cited alongside, same era.
Ground-truth, whose truth? – examining the challenges with annotating toxic text datasets
Kofi Arhin, Ioana Baldini, Dennis Wei, Karthikeyan Natesan Ramamurthy, and Moninder Singh. 2021 · 2021
Cited alongside, same era.
We need to consider disagreement in evaluation
Valerio Basile, Michael Fell, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, Massimo Poesio, and Alexandra Uma. 2021 · 2021
Cited alongside, same era.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022 · 2022
Later among the works it cites.
Detecting unintended social bias in toxic language datasets
Nihar Sahoo, Himanshu Gupta, and Pushpak Bhattacharyya. 2022 · 2022
Later among the works it cites.
Annotators with attitudes: How annotator beliefs and identities bias toxic language detection
Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A. Smith. 2022 · 2022
Later among the works it cites.
Identifying sociotechnical harms of algorithmic systems: Scoping a taxonomy for harm reduction
Renee Shelby, Shalaleh Rismani, Kathryn Henne, Ajung Moon, Negar Rostamzadeh, Paul Nicholas, YILLA-AKBARI N’MAH, Jess Gallegos, Andrew Smart, and GURLEEN VIRK. 2022 · 2022
Later among the works it cites.
Why so toxic? measuring and triggering toxic behavior in open-domain chatbots
Wai Man Si, Michael Backes, Jeremy Blackburn, Emiliano De Cristofaro, Gianluca Stringhini, Savvas Zannettou, and Yang Zhang. 2022 · 2022
Later among the works it cites.
LaMDA: Language models for dialog applications
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. 2022 · 2022
Later among the works it cites.
Toxicity detection sensitive to conversational context
Alexandros Xenos, John Pavlopoulos, Ion Androutsopoulos, Lucas Dixon, Jeffrey Sorensen, and Léo Laugier. 2022 · 2022
Later among the works it cites.
DICES Dataset: Diversity in conversational AI evaluation for safety
Lora Aroyo, Alex S. Taylor, Mark Díaz, Christopher Michael Homan, Alicia Parrish, Greg Serapio-García, Vinodkumar Prabhakaran, and Ding Wang. 2023 · 2023
Closest in time.
A drop of ink may make a million think: The spread of false information in large language models
Ning Bian, Peilin Liu, Xianpei Han, Hongyu Lin, Yaojie Lu, Ben He, and Le Sun. 2023 · 2023
Closest in time.
Deep reinforcement learning from human preferences
Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2023 · 2023
Closest in time.
You are what you annotate: Towards better models through annotator representations
Naihao Deng, Xinliang Zhang, Siyang Liu, Winston Wu, Lu Wang, and Rada Mihalcea. 2023 · 2023
Closest in time.
Google. 2023 · 2023
Closest in time.
Christopher Homan, Greg Serapio-García, Lora Aroyo, Mark Díaz, Alicia Parrish, Vinodkumar Prabhakaran, Alex S. Taylor, and Ding Wang. 2023 · 2023
Closest in time.
Is ChatGPT better than human annotators? potential and limitations of chatgpt in explaining implicit hate speech
Fan Huang, Haewoon Kwak, and Jisun An. 2023 · 2023
Closest in time.
Can demographic factors improve text classification? revisiting demographic adaptation in the age of transformers
Chia-Chien Hung, Anne Lauscher, Dirk Hovy, Simone Paolo Ponzetto, and Goran Glavaš. 2023 · 2023
Closest in time.
OpenAI. 2023 · 2023
Closest in time.
The ecological fallacy in annotation: Modeling human label variation goes beyond sociodemographics
Matthias Orlikowski, Paul Röttger, Philipp Cimiano, and Dirk Hovy. 2023 · 2023
Closest in time.
When do annotator demographics matter? measuring the influence of annotator demographics with the POPQUORN dataset
Jiaxin Pei and David Jurgens. 2023 · 2023
Closest in time.
Whose opinions do language models reflect?
Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023 · 2023
Closest in time.
Alpaca: A strong, replicable instruction-following model
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023 · 2023
Closest in time.
LLaMA: Open and efficient foundation language models
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023 · 2023
Closest in time.
Disagreement matters: Preserving label diversity by jointly modeling item and annotator label distributions with DisCo
Tharindu Cyril Weerasooriya, Alexander Ororbia, Raj Bhensadadia, Ashiqur KhudaBukhsh, and Christopher Homan. 2023b · 2023
Closest in time.
FACT SHEET: President Biden Issues Executive Order on Safe, Secure, and Trustworthy Artificial Intelligence
White House. 2023 · 2023
Closest in time.