Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognition

Understand

We introduce a new cross-modal fusion technique designed for generative error correction in automatic speech recognition (ASR).

• Our methodology leverages both acoustic information and external linguistic representations to generate accurate speech transcription contexts.
• This marks a step towards a fresh paradigm in generative error correction within the realm of n-best hypotheses.
• Unlike the existing ranking-based rescoring methods, our approach adeptly uses distinct initialization techniques and parameter-efficient algorithms to boost ASR performance derived from pre-trained speech and text models.