Are LLMs Better than Reported? Detecting Label Errors and Mitigating Their Effect on Model Performance

Self: Learning to filter noisy labels with self-ensembling

Original

Duc Tam Nguyen, Chaithanya Kumar Mummadi, Thi-Phuong-Nhung Ngo, Thi Hoai Phuong Nguyen, Laura Beggel, and Thomas Brox. 2019 · 1910

Earlier work this paper cites.

On faithfulness and factuality in abstractive summarization

Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan T. McDonald. 2020 · 1919

Earlier work this paper cites.

The use of confidence or fiducial limits illustrated in the case of the binomial

C. J. Clopper and E. S. Pearson. 1934 · 1934

Earlier work this paper cites.

Estimating the reliability, systematic error, and random error of interval data

Klaus Krippendorff. 1970 · 1970

Earlier work this paper cites.

Measuring nominal scale agreement among many raters

Joseph L. Fleiss. 1971 · 1971

Earlier work this paper cites.

A decision-theoretic generalization of on-line learning and an application to boosting

Yoav Freund and Robert E Schapire. 1997 · 1997

Earlier work this paper cites.

Scaling laws for neural language models

Original

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020 · 2001

Earlier work this paper cites.

Identifying mislabeled data using the area under the margin ranking

Original

Geoff Pleiss, Tianyi Zhang, Ethan R. Elenberg, and Kilian Q. Weinberger. 2020 · 2001

Earlier work this paper cites.

Ensemble methods in machine learning

Thomas G. Dietterich. 2007 · 2007

Earlier work this paper cites.

Cheap and fast – but is it good? evaluating non-expert annotations for natural language tasks

Rion Snow, Brendan T. O’Connor, Dan Jurafsky, and A. Ng. 2008 · 2008

Earlier work this paper cites.

An extended model of natural logic

Bill MacCartney and Christopher D. Manning. 2009 · 2009

Earlier work this paper cites.

Learning to combine discriminative classifiers: confidence based

Chi-Hoon Lee. 2010 · 2010

Earlier work this paper cites.

Translational biomarker discovery in clinical metabolomics: an introductory tutorial

Jianguo Xia, David I. Broadhurst, Michael Wilson, and David Scott Wishart. 2012 · 2012

Earlier work this paper cites.

Quality control in crowdsourcing systems: Issues and directions

Mohammad Allahbakhsh, Boualem Benatallah, Aleksandar Ignjatovic, Hamid Reza Motahari-Nezhad, Elisa Bertino, and Schahram Dustdar. 2013 · 2013

Earlier work this paper cites.

An analysis of human factors and label accuracy in crowdsourcing relevance judgments

Gabriella Kazai, Jaap Kamps, and Natasa Milic-Frayling. 2013 · 2013

Earlier work this paper cites.

Investigating the disagreement between clinicians’ ratings of patients in icus

Simon Rogers, Derek H. Sleeman, and John Kinsella. 2013 · 2013

Earlier work this paper cites.

Classification in the presence of label noise: A survey

Benoît Frénay and Michel Verleysen. 2014 · 2014

Earlier work this paper cites.

SQuAD: 100,000+ questions for machine comprehension of text

Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016 · 2016

Earlier work this paper cites.

Rethinking the inception architecture for computer vision

Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. 2016 · 2016

Earlier work this paper cites.

Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization

Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018 · 2018

Earlier work this paper cites.

FEVER: a large-scale dataset for fact extraction and VERification

James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018 · 2018

Earlier work this paper cites.

A broad-coverage challenge corpus for sentence understanding through inference

Adina Williams, Nikita Nangia, and Samuel Bowman. 2018 · 2018

Earlier work this paper cites.

mixup: Beyond empirical risk minimization

Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. 2018 · 2018

Earlier work this paper cites.

An MTurk crisis? shifts in data quality and the impact on study results

Michael Chmielewski and Sarah C. Kucker. 2019 · 2019

Earlier work this paper cites.

BERT: pre-training of deep bidirectional transformers for language understanding

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 · 2019

Earlier work this paper cites.

Wizard of wikipedia: Knowledge-powered conversational agents

Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2019 · 2019

Earlier work this paper cites.

Confident learning: Estimating uncertainty in dataset labels

Curtis G. Northcutt, Lu Jiang, and Isaac L. Chuang. 2019 · 2019

Earlier work this paper cites.

GLUE: A multi-task benchmark and analysis platform for natural language understanding

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019 · 2019

Earlier work this paper cites.

PAWS: Paraphrase adversaries from word scrambling

Yuan Zhang, Jason Baldridge, and Luheng He. 2019 · 2019

Earlier work this paper cites.

Language models are few-shot learners

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020 · 2020

Cited alongside, same era.

Understanding the tradeoff between cost and quality of expert annotations for keyphrase extraction

Hung Chau, Saeid Balaneshin, Kai Liu, and Ondrej Linda. 2020 · 2020

Cited alongside, same era.

Inaccurate labels in weakly-supervised deep learning: Automatic identification and correction and their impact on classification performance

Degan Hao, Lei Zhang, Jules H. Sumkin, Aly A. Mohamed, and Shandong Wu. 2020 · 2020

Cited alongside, same era.

The shape of and solutions to the MTurk quality crisis

Ryan Kennedy, Scott Clifford, Tyler Burleigh, Philip D Waggoner, Ryan Jewell, and Nicholas JG Winter. 2020 · 2020

Cited alongside, same era.

Research on data quality control of crowdsourcing annotation: A survey

Self-refine: Iterative refinement with self-feedback

Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023 · 2023

Later among the works it cites.

GPT-4 technical report

Original

OpenAI. 2023 · 2023

Later among the works it cites.

Predictive performance of multi-model ensemble forecasts of covid-19 across european nations

Katharine Sherratt, Hugo Gruson, Rok Grah, Helen Johnson, Rene Niehus, Bastian Prasse, and et al. 2023 · 2023

Later among the works it cites.

Beyond the imitation game: Quantifying and extrapolating the capabilities of language models

Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ambrose Slone, Ameet Rahane, Anantharaman S. Iyer, Anders Andreassen, and et al. 2023 · 2023

Later among the works it cites.