Fast and High-Quality Auto-Regressive Speech Synthesis via Speculative Decoding

Understand

The auto-regressive architecture, like GPTs, is widely used in modern Text-to-Speech (TTS) systems.

• However, it incurs substantial inference time, particularly due to the challenges in the next-token prediction posed by lengthy sequences of speech tokens.
• In this work, we introduce VADUSA, one of the first approaches to accelerate auto-regressive TTS through speculative decoding.
• Our results show that VADUSA not only significantly improves inference speed but also enhances performance by incorporating draft heads to predict future speech content auto-regressively.