Why Knowledge Distillation Works in Generative Models: A Minimal Working Explanation

Understand

Knowledge distillation (KD) is a core component in the training and deployment of modern generative models, particularly large language models (LLMs).

• While its empirical benefits are well documented -- enabling smaller student models to emulate the performance of much larger teachers -- the underlying mechanisms by which KD improves generative quality remain poorly understood.
• In this work, we present a minimal working explanation of KD in generative modeling.
• Using a controlled simulation with mixtures of Gaussians, we demonstrate that distillation induces a trade-off between precision and recall in the student model.