Learning by Surprise: Adaptive Mitigation of Model Collapse in Large Language Models

Understand

As AI-generated content increasingly populates the web, generative AI models are at growing risk of being trained on their own outputs, a process known as AI autophagy.

• This feedback loop has been shown to induce model collapse, typically characterized by a loss of diversity in generated content.
• However, existing work offers a limited understanding of this phenomenon and relies on mitigation strategies that assume access to human-authored data.
• In this paper, we conduct extensive simulations across multiple datasets and LLMs to address key gaps in the study of model collapse.