Revisiting Adversarial Robustness Distillation: Robust Soft Labels Make Student Better

Understand

Adversarial training is one effective approach for training robust deep neural networks against adversarial attacks.

• While being able to bring reliable robustness, adversarial training (AT) methods in general favor high capacity models, i.e., the larger the model the better the robustness.
• This tends to limit their effectiveness on small models, which are more preferable in scenarios where storage or computing resources are very limited (e.g., mobile devices).
• In this paper, we leverage the concept of knowledge distillation to improve the robustness of small models by distilling from adversarially trained large models.