StyleS2ST: Zero-shot Style Transfer for Direct Speech-to-speech Translation

Understand

Direct speech-to-speech translation (S2ST) has gradually become popular as it has many advantages compared with cascade S2ST.

• However, current research mainly focuses on the accuracy of semantic translation and ignores the speech style transfer from a source language to a target language.
• The lack of high-fidelity expressive parallel data makes such style transfer challenging, especially in more practical zero-shot scenarios.
• To solve this problem, we first build a parallel corpus using a multi-lingual multi-speaker text-to-speech synthesis (TTS) system and then propose the StyleS2ST model with cross-lingual speech style transfer ability based on a style adaptor on a direct S2ST system framework.