ALP-KD: Attention-Based Layer Projection for Knowledge Distillation

Understand

Knowledge distillation is considered as a training and compression strategy in which two neural networks, namely a teacher and a student, are coupled together during training.

• The teacher network is supposed to be a trustworthy predictor and the student tries to mimic its predictions.
• Usually, a student with a lighter architecture is selected so we can achieve compression and yet deliver high-quality results.
• In such a setting, distillation only happens for final predictions whereas the student could also benefit from teacher's supervision for internal components.