Whisper-GPT -- Continuous Discrete Hybrid Representation Language Models For Speech And Music

Understand

We propose WHISPER-GPT: A generative large language model (LLM) for speech and music that allows us to work with continuous audio representations and discrete tokens simultaneously as part of a single architecture.

• There has been a huge surge in generative audio, speech, and music models that utilize discrete audio tokens derived from neural compression algorithms, e.g.
• ENCODEC.
• However, one of the major drawbacks of this approach is handling the context length.