Textless Unit-to-Unit training for Many-to-Many Multilingual Speech-to-Speech Translation

Understand

This paper proposes a textless training method for many-to-many multilingual speech-to-speech translation that can also benefit the transfer of pre-trained knowledge to text-based systems, text-to-speech synthesis and text-to-speech translation.

• To this end, we represent multilingual speech with speech units that are the discretized representations of speech features derived from a self-supervised speech model.
• By treating the speech units as pseudo-text, we can focus on the linguistic content of the speech, which can be easily associated with both speech and text modalities at the phonetic level information.
• By setting both the inputs and outputs of our learning problem as speech units, we propose to train an encoder-decoder model in a many-to-many spoken language translation setting, namely Unit-to-Unit Translation (UTUT).