Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLM

Understand

We present Spectron, a novel approach to adapting pre-trained large language models (LLMs) to perform spoken question answering (QA) and speech continuation.

• By endowing the LLM with a pre-trained speech encoder, our model becomes able to take speech inputs and generate speech outputs.
• The entire system is trained end-to-end and operates directly on spectrograms, simplifying our architecture.
• Key to our approach is a training objective that jointly supervises speech recognition, text continuation, and speech synthesis using only paired speech-text pairs, enabling a `cross-modal' chain-of-thought within a single decoding pass.