On the Transferability of Whisper-based Representations for "In-the-Wild" Cross-Task Downstream Speech Applications

Understand

Large self-supervised pre-trained speech models have achieved remarkable success across various speech-processing tasks.

• The self-supervised training of these models leads to universal speech representations that can be used for different downstream tasks, ranging from automatic speech recognition (ASR) to speaker identification.
• Recently, Whisper, a transformer-based model was proposed and trained on large amount of weakly supervised data for ASR; it outperformed several state-of-the-art self-supervised models.
• Given the superiority of Whisper for ASR, in this paper we explore the transferability of the representation for four other speech tasks in SUPERB benchmark.