Multilingual DistilWhisper: Efficient Distillation of Multi-task Speech Models via Language-Specific Experts

Understand

Whisper is a multitask and multilingual speech model covering 99 languages.

• It yields commendable automatic speech recognition (ASR) results in a subset of its covered languages, but the model still underperforms on a non-negligible number of under-represented languages, a problem exacerbated in smaller model versions.
• In this work, we propose DistilWhisper, an approach able to bridge the performance gap in ASR for these languages while retaining the advantages of multitask and multilingual capabilities.
• Our approach involves two key strategies: lightweight modular ASR fine-tuning of whisper-small using language-specific experts, and knowledge distillation from whisper-large-v2.