Towards Understanding Knowledge Distillation

Understand

Knowledge distillation, i.e., one classifier being trained on the outputs of another classifier, is an empirically very successful technique for knowledge transfer between classifiers.

• It has even been observed that classifiers learn much faster and more reliably if trained with the outputs of another classifier as soft labels, instead of from ground truth data.
• So far, however, there is no satisfactory theoretical explanation of this phenomenon.
• In this work, we provide the first insights into the working mechanisms of distillation by studying the special case of linear and deep linear classifiers.