Understanding and Improving Knowledge Distillation

Understand

Knowledge Distillation (KD) is a model-agnostic technique to improve model quality while having a fixed capacity budget.

• It is a commonly used technique for model compression, where a larger capacity teacher model with better quality is used to train a more compact student model with better inference efficiency.
• Through distillation, one hopes to benefit from student's compactness, without sacrificing too much on model quality.
• Despite the large success of knowledge distillation, better understanding of how it benefits student model's training dynamics remains under-explored.