Low-Resource Domain Adaptation for Speech LLMs via Text-Only Fine-Tuning

Understand

Recent advances in automatic speech recognition (ASR) have combined speech encoders with large language models (LLMs) through projection, forming Speech LLMs with strong performance.

• However, adapting them to new domains remains challenging, especially in low-resource settings where paired speech-text data is scarce.
• We propose a text-only fine-tuning strategy for Speech LLMs using unpaired target-domain text without requiring additional audio.
• To preserve speech-text alignment, we introduce a real-time evaluation mechanism during fine-tuning.