Knowledge Distillation from Internal Representations

Understand

Knowledge distillation is typically conducted by training a small model (the student) to mimic a large and cumbersome model (the teacher).

• The idea is to compress the knowledge from the teacher by using its output probabilities as soft-labels to optimize the student.
• However, when the teacher is considerably large, there is no guarantee that the internal knowledge of the teacher will be transferred into the student; even if the student closely matches the soft-labels, its internal representations may be considerably different.
• This internal mismatch can undermine the generalization capabilities originally intended to be transferred from the teacher to the student.