Better Teacher Better Student: Dynamic Prior Knowledge for Knowledge Distillation

Understand

Knowledge distillation (KD) has shown very promising capabilities in transferring learning representations from large models (teachers) to small models (students).

• However, as the capacity gap between students and teachers becomes larger, existing KD methods fail to achieve better results.
• Our work shows that the `prior knowledge' is vital to KD, especially when applying large teachers.
• Particularly, we propose the dynamic prior knowledge (DPK), which integrates part of teacher's features as the prior knowledge before the feature distillation.