Predicting Multi-Codebook Vector Quantization Indexes for Knowledge Distillation

Understand

Knowledge distillation(KD) is a common approach to improve model performance in automatic speech recognition (ASR), where a student model is trained to imitate the output behaviour of a teacher model.

• However, traditional KD methods suffer from teacher label storage issue, especially when the training corpora are large.
• Although on-the-fly teacher label generation tackles this issue, the training speed is significantly slower as the teacher model has to be evaluated every batch.
• In this paper, we reformulate the generation of teacher label as a codec problem.