SeamlessExpressiveLM: Speech Language Model for Expressive Speech-to-Speech Translation with Chain-of-Thought

Understand

Expressive speech-to-speech translation (S2ST) is a key research topic in seamless communication, which focuses on the preservation of semantics and speaker vocal style in translated speech.

• Early works synthesized speaker style aligned speech in order to directly learn the mapping from speech to target speech spectrogram.
• Without reliance on style aligned data, recent studies leverage the advances of language modeling (LM) and build cascaded LMs on semantic and acoustic tokens.
• This work proposes SeamlessExpressiveLM, a single speech language model for expressive S2ST.