Fetching the paper…
Reading the bibliography…
What happens when generative machine learning models are pretrained on web-scale datasets containing data generated by earlier models? Some prior work warns of "model collapse" as the web is overwhelmed by synthetic data; other work suggests the problem can be contained (i.e.
Scikit-learn: Machine learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., and Duchesnay, E · 2011
Earlier work this paper cites.
Language models are few-shot learners
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D · 2020
Earlier work this paper cites.
Self-distillation amplifies regularization in hilbert space
Mobahi, H., Farajtabar, M., and Bartlett, P · 2020
Earlier work this paper cites.
On the opportunities and risks of foundation models, 2022
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., Brynjolfsson, E., Buch, S., Card, D., Castellon, R., Chatterji, N., Chen, A., Creel, K., Davis, J. Q., Demszky, D., Donahue, C., Doumbouya, M., Durmus, E., Ermon, S., Etchemendy, J., Ethayarajh, K., Fei-Fei, L., Finn, C., Gale, T., Gillespie, L., Goel, K., Goodman, N., Grossman, S., Guha, N., Hashimoto, T., Henderson, P., Hewitt, J., Ho, D. E., Hong, J., Hsu, K., Huang, J., Icard, T., Jain, S., Jurafsky, D., Kalluri, P., Karamcheti, S., Keeling, G., Khani, F., Khattab, O., Koh, P. W., Krass, M., Krishna, R., Kuditipudi, R., Kumar, A., Ladhak, F., Lee, M., Lee, T., Leskovec, J., Levent, I., Li, X. L., Li, X., Ma, T., Malik, A., Manning, C. D., Mirchandani, S., Mitchell, E., Munyikwa, Z., Nair, S., Narayan, A., Narayanan, D., Newman, B., Nie, A., Niebles, J. C., Nilforoshan, H., Nyarko, J., Ogut, G., Orr, L., Papadimitriou, I., Park, J. S., Piech, C., Portelance, E., Potts, C., Raghunathan, A., Reich, R., Ren, H., Rong, F., Roohani, Y., Ruiz, C., Ryan, J., Ré, C., Sadigh, D., Sagawa, S., Santhanam, K., Shih, A., Srinivasan, K., Tamkin, A., Taori, R., Thomas, A. W., Tramèr, F., Wang, R. E., Wang, W., Wu, B., Wu, J., Wu, Y., Xie, S. M., Yasunaga, M., You, J., Zaharia, M., Zhang, M., Zhang, T., Zhang, X., Zhang, Y., Zheng, L., Zhou, K., and Liang, P · 2022
Earlier work this paper cites.
Nepotistically trained generative-ai models collapse
Bohacek, M. and Farid, H · 2023
Earlier work this paper cites.
Large language models suffer from their own output: An analysis of the self-consuming training loop
Briesch, M., Sobania, D., and Rothlauf, F · 2023
Earlier work this paper cites.
Tinystories: How small can language models be and still speak coherent english?
Eldan, R. and Li, Y · 2023
Earlier work this paper cites.
Will large-scale generative models corrupt future datasets?
Hataya, R., Bao, H., and Arai, H · 2023
Earlier work this paper cites.
Lee, A., Miranda, B., Sundar, S., and Koyejo, S · 2023
Earlier work this paper cites.
Martínez, G., Watson, L., Reviriego, P., Hernández, J. A., Juarez, M., and Sarkar, R · 2023
Earlier work this paper cites.
Towards understanding the interplay of generative artificial intelligence and the internet
Martínez, G., Watson, L., Reviriego, P., Hernández, J. A., Juarez, M., and Sarkar, R · 2023
Earlier work this paper cites.
The curse of recursion: Training on generated data makes models forget
Shumailov, I., Shumaylov, Z., Zhao, Y., Gal, Y., Papernot, N., and Anderson, R. J · 2023
Earlier work this paper cites.
Data feedback loops: Model-driven amplification of dataset biases
Taori, R. and Hashimoto, T · 2023
Earlier work this paper cites.
Star: self-taught reasoner bootstrapping reasoning with reasoning
Zelikman, E., Wu, Y., Mu, J., and Goodman, N. D · 2023
Cited alongside, same era.
A survey on data selection for language models, 2024
Albalak, A., Elazar, Y., Xie, S. M., Longpre, S., Lambert, N., Wang, X., Muennighoff, N., Hou, B., Pan, L., Jeong, H., Raffel, C., Chang, S., Hashimoto, T., and Wang, W. Y · 2024
Cited alongside, same era.
Self-consuming generative models go MAD
Alemohammad, S., Casco-Rodriguez, J., Luzi, L., Humayun, A. I., Babaei, H., LeJeune, D., Siahkoohi, A., and Baraniuk, R · 2024
Cited alongside, same era.
openai now generates about 100 billion words per day., Feb 2024
Altman, S · 2024
Cited alongside, same era.
On the stability of iterative retraining of generative models on their own data
Bertrand, Q., Bose, A. J., Duplessis, A., Jiralerspong, M., and Gidel, G · 2024
Cited alongside, same era.
Does writing with language models reduce content diversity?
Padmakumar, V. and He, H · 2024
Closest in time.
The fineweb datasets: Decanting the web for the finest text data at scale
Penedo, G., Kydlíček, H., allal, L. B., Lozhkov, A., Mitchell, M., Raffel, C., Werra, L. V., and Wolf, T · 2024
Closest in time.
Artificial intelligence index report 2024
Perrault, R. and Clark, J · 2024
Closest in time.
Open problems in technical ai governance, 2024
Reuel, A., Bucknall, B., Casper, S., Fist, T., Soder, L., Aarne, O., Hammond, L., Ibrahim, L., Chan, A., Wills, P., Anderljung, M., Garfinkel, B., Heim, L., Trask, A., Mukobi, G., Schaeffer, R., Baker, M., Hooker, S., Solaiman, I., Luccioni, A. S., Rajkumar, N., Moës, N., Ladish, J., Guha, N., Newman, J., Bengio, Y., South, T., Pentland, A., Koyejo, S., Kochenderfer, M. J., and Trager, R · 2024
Closest in time.
How to train data-efficient llms
Sachdeva, N., Coleman, B., Kang, W.-C., Ni, J., Hong, L., Chi, E. H., Caverlee, J., McAuley, J. J., and Cheng, D. Z · 2024
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Would deep generative models amplify bias in future models?
Chen, T., Hirota, Y., Otani, M., Garcia, N., and Nakashima, Y · 2024
Cited alongside, same era.
Universality of the π 2 / 6 \pi^{2}/6 pathway in avoiding model collapse
Dey, A. and Donoho, D · 2024
Cited alongside, same era.
Is model collapse inevitable? breaking the curse of recursion by accumulating real and synthetic data, 2024
Gerstgrasser, M., Schaeffer, R., Dey, A., Rafailov, R., Korbak, T., Sleight, H., Agrawal, R., Hughes, J., Pai, D. B., Gromov, A., Roberts, D., Yang, D., Donoho, D. L., and Koyejo, S · 2024
Cited alongside, same era.
Self-correcting self-consuming loops for generative model training, 2024
Gillman, N., Freeman, M., Aggarwal, D., Hsu, C.-H., Luo, C., Tian, Y., and Sun, C · 2024
Cited alongside, same era.
The curious decline of linguistic diversity: Training language models on synthetic text
Guo, Y., Shang, G., Vazirgiannis, M., and Clavel, C · 2024
Cited alongside, same era.
Scaling laws for learning with real and surrogate data, 2024
Jain, A., Montanari, A., and Sasoglu, E · 2024
Cited alongside, same era.
Position: On the societal impact of open foundation models
Kapoor, S., Bommasani, R., Klyman, K., Longpre, S., Ramaswami, A., Cihon, P., Hopkins, A., Bankston, K., Biderman, S., Bogen, M., et al · 2024
Cited alongside, same era.
Closest in time.
How bad is training on synthetic data? a statistical analysis of language model collapse, 2024
Seddik, M. E. A., Chen, S.-W., Hayou, S., Youssef, P., and Debbah, M · 2024
Closest in time.
Ai models collapse when trained on recursively generated data
Shumailov, I., Shumaylov, Z., Zhao, Y., Papernot, N., Anderson, R., and Gal, Y · 2024
Closest in time.
Gemma 2: Improving open language models at a practical size, 2024
Team, G., Riviere, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahriari, B., Ramé, A., Ferret, J., Liu, P., Tafti, P., Friesen, A., Casbon, M., Ramos, S., Kumar, R., Lan, C. L., Jerome, S., Tsitsulin, A., Vieillard, N., Stanczyk, P., Girgin, S., Momchev, N., Hoffman, M., Thakoor, S., Grill, J.-B., Neyshabur, B., Bachem, O., Walton, A., Severyn, A., Parrish, A., Ahmad, A., Hutchison, A., Abdagic, A., Carl, A., Shen, A., Brock, A., Coenen, A., Laforge, A., Paterson, A., Bastian, B., Piot, B., Wu, B., Royal, B., Chen, C., Kumar, C., Perry, C., Welty, C., Choquette-Choo, C. A., Sinopalnikov, D., Weinberger, D., Vijaykumar, D., Rogozińska, D., Herbison, D., Bandy, E., Wang, E., Noland, E., Moreira, E., Senter, E., Eltyshev, E., Visin, F., Rasskin, G., Wei, G., Cameron, G., Martins, G., Hashemi, H., Klimczak-Plucińska, H., Batra, H., Dhand, H., Nardini, I., Mein, J., Zhou, J., Svensson, J., Stanway, J., Chan, J., Zhou, J. P., Carrasqueira, J., Iljazi, J., Becker, J., Fernandez, J., van Amersfoort, J., Gordon, J., Lipschultz, J., Newlan, J., yeong Ji, J., Mohamed, K., Badola, K., Black, K., Millican, K., McDonell, K., Nguyen, K., Sodhia, K., Greene, K., Sjoesund, L. L., Usui, L., Sifre, L., Heuermann, L., Lago, L., McNealus, L., Soares, L. B., Kilpatrick, L., Dixon, L., Martins, L., Reid, M., Singh, M., Iverson, M., Görner, M., Velloso, M., Wirth, M., Davidow, M., Miller, M., Rahtz, M., Watson, M., Risdal, M., Kazemi, M., Moynihan, M., Zhang, M., Kahng, M., Park, M., Rahman, M., Khatwani, M., Dao, N., Bardoliwalla, N., Devanathan, N., Dumai, N., Chauhan, N., Wahltinez, O., Botarda, P., Barnes, P., Barham, P., Michel, P., Jin, P., Georgiev, P., Culliton, P., Kuppala, P., Comanescu, R., Merhej, R., Jana, R., Rokni, R. A., Agarwal, R., Mullins, R., Saadat, S., Carthy, S. M., Perrin, S., Arnold, S. M. R., Krause, S., Dai, S., Garg, S., Sheth, S., Ronstrom, S., Chan, S., Jordan, T., Yu, T., Eccles, T., Hennigan, T., Kocisky, T., Doshi, T., Jain, V., Yadav, V., Meshram, V., Dharmadhikari, V., Barkley, W., Wei, W., Ye, W., Han, W., Kwon, W., Xu, X., Shen, Z., Gong, Z., Wei, Z., Cotruta, V., Kirk, P., Rao, A., Giang, M., Peran, L., Warkentin, T., Collins, E., Barral, J., Ghahramani, Z., Hadsell, R., Sculley, D., Banks, J., Dragan, A., Petrov, S., Vinyals, O., Dean, J., Hassabis, D., Kavukcuoglu, K., Farabet, C., Buchatskaya, E., Borgeaud, S., Fiedel, N., Joulin, A., Kenealy, K., Dadashi, R., and Andreev, A · 2024
Closest in time.
A mathematical model of the hidden feedback loop effect in machine learning systems
Veprikov, A., Afanasiev, A., and Khritankov, A · 2024
Closest in time.
Helpsteer2: Open-source dataset for training top-performing reward models, 2024
Wang, Z., Dong, Y., Delalleau, O., Zeng, J., Shen, G., Egert, D., Zhang, J. J., Sreedhar, M. N., and Kuchaiev, O · 2024
Closest in time.
QuRating: Selecting high-quality data for training language models
Wettig, A., Gupta, A., Malik, S., and Chen, D · 2024
Closest in time.
Synthetic continued pretraining, 2024
Yang, Z., Band, N., Li, S., Candès, E., and Hashimoto, T · 2024
Closest in time.
The internet contains an increasing amount of ai generated data…
Appenzeller, G · 2025
Closest in time.