ViTKD: Practical Guidelines for ViT feature knowledge distillation

Understand

Knowledge Distillation (KD) for Convolutional Neural Network (CNN) is extensively studied as a way to boost the performance of a small model.

• Recently, Vision Transformer (ViT) has achieved great success on many computer vision tasks and KD for ViT is also desired.
• However, besides the output logit-based KD, other feature-based KD methods for CNNs cannot be directly applied to ViT due to the huge structure gap.
• In this paper, we explore the way of feature-based distillation for ViT.