Fetching the paper…
Reading the bibliography…
The phenomenon of model collapse, introduced in (Shumailov et al., 2023), refers to the deterioration in performance that occurs when new models are trained on synthetic data generated from previously trained models.
Finite markov chains , volume 26
John G Kemeny, J Laurie Snell, et al · 1969
Earlier work this paper cites.
Inequalities for the l1 deviation of the empirical distribution
Tsachy Weissman, Erik Ordentlich, Gadiel Seroussi, Sergio Verdu, and Marcelo J Weinberger · 2003
Earlier work this paper cites.
Concentration inequalities for the empirical distribution of discrete distributions: beyond the method of types
Jay Mardia, Jiantao Jiao, Ervin Tánczos, Robert D Nowak, and Tsachy Weissman · 2020
Earlier work this paper cites.
How many pretraining tasks are needed for in-context learning of linear regression?
Jingfeng Wu, Difan Zou, Zixiang Chen, Vladimir Braverman, Quanquan Gu, and Peter L Bartlett · 2020
Earlier work this paper cites.
Laion-5b: An open large-scale dataset for training next generation image-text models, 2022
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev · 2022
Earlier work this paper cites.
Self-consuming generative models go mad, 2023
Sina Alemohammad, Josue Casco-Rodriguez, Lorenzo Luzi, Ahmed Imtiaz Humayun, Hossein Babaei, Daniel LeJeune, Ali Siahkoohi, and Richard G. Baraniuk · 2023
Earlier work this paper cites.
Nepotistically trained generative-ai models collapse, 2023
Matyas Bohacek and Hany Farid · 2023
Cited alongside, same era.
Large language models suffer from their own output: An analysis of the self-consuming training loop, 2023
Martin Briesch, Dominik Sobania, and Franz Rothlauf · 2023
Cited alongside, same era.
Are large language models a threat to digital public goods? evidence from activity on stack overflow, 2023
Maria del Rio-Chanona, Nadzeya Laurentsyeva, and Johannes Wachs · 2023
Cited alongside, same era.
Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al · 2023
Cited alongside, same era.
The curious decline of linguistic diversity: Training language models on synthetic text, 2023
Yanzhu Guo, Guokan Shang, Michalis Vazirgiannis, and Chloé Clavel · 2023
Cited alongside, same era.
The curse of recursion: Training on generated data makes models forget, 2023
Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Anderson · 2023
Later among the works it cites.
Cosmopedia, 2024
Loubna Ben Allal, Anton Lozhkov, Guilherme Penedo, Thomas Wolf, and Leandro von Werra · 2024
Closest in time.
Self-play fine-tuning converts weak language models to strong language models
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu · 2024
Closest in time.
Towards theoretical understandings of self-consuming generative models, 2024
Shi Fu, Sen Zhang, Yingjie Wang, Xinmei Tian, and Dacheng Tao · 2024
Closest in time.
Gpt-4 technical report, 2024
OpenAI · 2024
Closest in time.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Model collapse demystified: The case of regression, 2024a
Elvis Dohmatob, Yunzhen Feng, and Julia Kempe
Cited in the paper.
A tale of tails: Model collapse as a change of scaling laws, 2024b
Elvis Dohmatob, Yunzhen Feng, Pu Yang, Francois Charton, and Julia Kempe
Cited in the paper.
Combining generative artificial intelligence (ai) and the internet: Heading towards evolution or degradation?, 2023a
Gonzalo Martínez, Lauren Watson, Pedro Reviriego, José Alberto Hernández, Marc Juarez, and Rik Sarkar
Cited in the paper.
Towards understanding the interplay of generative artificial intelligence and the internet, 2023b
Gonzalo Martínez, Lauren Watson, Pedro Reviriego, José Alberto Hernández, Marc Juarez, and Rik Sarkar
Cited in the paper.