An Embarrassingly Simple Approach for Knowledge Distillation

Understand

Knowledge Distillation (KD) aims at improving the performance of a low-capacity student model by inheriting knowledge from a high-capacity teacher model.

• Previous KD methods typically train a student by minimizing a task-related loss and the KD loss simultaneously, using a pre-defined loss weight to balance these two terms.
• In this work, we propose to first transfer the backbone knowledge from a teacher to the student, and then only learn the task-head of the student network.
• Such a decomposition of the training process circumvents the need of choosing an appropriate loss weight, which is often difficult in practice, and thus makes it easier to apply to different datasets and tasks.