QKD: Quantization-aware Knowledge Distillation

Understand

Quantization and Knowledge distillation (KD) methods are widely used to reduce memory and power consumption of deep neural networks (DNNs), especially for resource-constrained edge devices.

• Although their combination is quite promising to meet these requirements, it may not work as desired.
• It is mainly because the regularization effect of KD further diminishes the already reduced representation power of a quantized model.
• To address this short-coming, we propose Quantization-aware Knowledge Distillation (QKD) wherein quantization and KD are care-fully coordinated in three phases.