Graph-based Knowledge Distillation by Multi-head Attention Network

Understand

Knowledge distillation (KD) is a technique to derive optimal performance from a small student network (SN) by distilling knowledge of a large teacher network (TN) and transferring the distilled knowledge to the small SN.

• Since a role of convolutional neural network (CNN) in KD is to embed a dataset so as to perform a given task well, it is very important to acquire knowledge that considers intra-data relations.
• Conventional KD methods have concentrated on distilling knowledge in data units.
• To our knowledge, any KD methods for distilling information in dataset units have not yet been proposed.