Aligning Speech to Languages to Enhance Code-switching Speech Recognition

Understand

Code-switching (CS) refers to the switching of languages within a speech signal and results in language confusion for automatic speech recognition (ASR).

• To address language confusion, we propose a language alignment loss (LAL) that aligns acoustic features to pseudo-language labels learned from the ASR decoder during ASR training.
• This approach enables frame-level language identification without the need for frame-level language annotations.
• To further tackle the complex token alternatives for language modeling in bilingual scenarios, we propose to employ large language models via a generative error correction method.