AMD: Automatic Multi-step Distillation of Large-scale Vision Models

Understand

Transformer-based architectures have become the de-facto standard models for diverse vision tasks owing to their superior performance.

• As the size of the models continues to scale up, model distillation becomes extremely important in various real applications, particularly on devices limited by computational resources.
• However, prevailing knowledge distillation methods exhibit diminished efficacy when confronted with a large capacity gap between the teacher and the student, e.g, 10x compression rate.
• In this paper, we present a novel approach named Automatic Multi-step Distillation (AMD) for large-scale vision model compression.