Knowledge Distillation with Adversarial Samples Supporting Decision Boundary

Understand

Many recent works on knowledge distillation have provided ways to transfer the knowledge of a trained network for improving the learning process of a new one, but finding a good technique for knowledge distillation is still an open problem.

• In this paper, we provide a new perspective based on a decision boundary, which is one of the most important component of a classifier.
• The generalization performance of a classifier is closely related to the adequacy of its decision boundary, so a good classifier bears a good decision boundary.
• Therefore, transferring information closely related to the decision boundary can be a good attempt for knowledge distillation.