Self-Distillation Amplifies Regularization in Hilbert Space

Understand

Knowledge distillation introduced in the deep learning context is a method to transfer knowledge from one architecture to another.

• In particular, when the architectures are identical, this is called self-distillation.
• The idea is to feed in predictions of the trained model as new target values for retraining (and iterate this loop possibly a few times).
• It has been empirically observed that the self-distilled model often achieves higher accuracy on held out data.