DQ-Whisper: Joint Distillation and Quantization for Efficient Multilingual Speech Recognition

Understand

As a popular multilingual and multitask pre-trained speech model, Whisper has the problem of curse of multilinguality.

• To enhance multilingual capabilities in small Whisper models, we propose DQ-Whisper, a novel joint distillation and quantization framework to compress Whisper for efficient inference.
• Firstly, we propose a novel dynamic matching distillation strategy.
• Then, a quantization-aware distillation framework is introduced to integrate quantization with distillation.