Adversarially Robust Distillation

Understand

Knowledge distillation is effective for producing small, high-performance neural networks for classification, but these small networks are vulnerable to adversarial attacks.

• This paper studies how adversarial robustness transfers from teacher to student during knowledge distillation.
• We find that a large amount of robustness may be inherited by the student even when distilled on only clean images.
• Second, we introduce Adversarially Robust Distillation (ARD) for distilling robustness onto student networks.