Data-Free Adversarial Distillation

Understand

Knowledge Distillation (KD) has made remarkable progress in the last few years and become a popular paradigm for model compression and knowledge transfer.

• However, almost all existing KD algorithms are data-driven, i.e., relying on a large amount of original training data or alternative data, which is usually unavailable in real-world scenarios.
• In this paper, we devote ourselves to this challenging problem and propose a novel adversarial distillation mechanism to craft a compact student model without any real-world data.
• We introduce a model discrepancy to quantificationally measure the difference between student and teacher models and construct an optimizable upper bound.