The Role of Masking for Efficient Supervised Knowledge Distillation of Vision Transformers

Understand

Knowledge distillation is an effective method for training lightweight vision models.

• However, acquiring teacher supervision for training samples is often costly, especially from large-scale models like vision transformers (ViTs).
• In this paper, we develop a simple framework to reduce the supervision cost of ViT distillation: masking out a fraction of input tokens given to the teacher.
• By masking input tokens, one can skip the computations associated with the masked tokens without requiring any change to teacher parameters or architecture.