TASTE: Text-Aligned Speech Tokenization and Embedding for Spoken Language Modeling

Understand

Recent efforts target spoken language models (SLMs) that not only listen but also speak for more natural human-LLM interaction.

• Joint speech-text modeling is a promising direction to achieve this.
• However, the effectiveness of recent speech tokens for joint modeling remains underexplored.
• To address this, we introduce Text-Aligned Speech Tokenization and Embedding (TASTE), a method that directly addresses the modality gap by aligning speech token with the corresponding text transcription during the tokenization stage.