Large Language Models Suffer From Their Own Output: An Analysis of the Self-Consuming Training Loop

Language models are few-shot learners

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020 · 1901

Earlier work this paper cites.

Bertscore: Evaluating text generation with bert

Original

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019 · 1904

Earlier work this paper cites.

Energy and policy considerations for deep learning in nlp

Original

Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019 · 1906

Earlier work this paper cites.

Binary codes capable of correcting deletions, insertions, and reversals

Vladimir I Levenshtein et al. 1966 · 1966

Earlier work this paper cites.

Bleu: a method for automatic evaluation of machine translation

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002 · 2002

Earlier work this paper cites.

Rouge: A package for automatic evaluation of summaries

Chin-Yew Lin. 2004 · 2004

Earlier work this paper cites.

Re-evaluating the role of bleu in machine translation research

Chris Callison-Burch, Miles Osborne, and Philipp Koehn. 2006 · 2006

Earlier work this paper cites.

Speech and language processing: An introduction to natural language processing, computational linguistics, and speech recognition

Dan Jurafsky and James H Martin. 2009 · 2009

Earlier work this paper cites.

Confronting the challenge of quality diversity

Justin K Pugh, Lisa B Soros, Paul A Szerlip, and Kenneth O Stanley. 2015 · 2015

Earlier work this paper cites.

Evaluating large language models trained on code

Original

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021 · 2021

Earlier work this paper cites.

Differentiable quality diversity

Matthew Fontaine and Stefanos Nikolaidis. 2021 · 2021

Earlier work this paper cites.

Large language models can self-improve

Original

Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. 2022 · 2022

Cited alongside, same era.

All the news that’s fit to fabricate: Ai-generated text as a tool of media misinformation

Sarah Kreps, R Miles McCain, and Miles Brundage. 2022 · 2022

Cited alongside, same era.

Choose your programming copilot: A comparison of the program synthesis performance of github copilot and genetic programming

Dominik Sobania, Martin Briesch, and Franz Rothlauf. 2022 · 2022

Cited alongside, same era.

Will we run out of data? an analysis of the limits of scaling datasets in machine learning

Original

Pablo Villalobos, Jaime Sevilla, Lennart Heim, Tamay Besiroglu, Marius Hobbhahn, and Anson Ho. 2022 · 2022

Cited alongside, same era.

Self-instruct: Aligning language model with self generated instructions

Do you trust chatgpt?–perceived credibility of human and ai-generated content

Original

Martin Huschens, Martin Briesch, Dominik Sobania, and Franz Rothlauf. 2023 · 2023

Closest in time.

Towards understanding the interplay of generative artificial intelligence and the internet

Original

Gonzalo Martínez, Lauren Watson, Pedro Reviriego, José Alberto Hernández, Marc Juarez, and Rik Sarkar. 2023 · 2023

Closest in time.

Can ai-generated text be reliably detected?

Original

Vinu Sankar Sadasivan, Aounon Kumar, Sriram Balasubramanian, Wenxiao Wang, and Soheil Feizi. 2023 · 2023

Closest in time.

The curse of recursion: Training on generated data makes models forget

Original

Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Anderson. 2023 · 2023

Closest in time.