RetrieverTTS: Modeling Decomposed Factors for Text-Based Speech Insertion

Understand

This paper proposes a new "decompose-and-edit" paradigm for the text-based speech insertion task that facilitates arbitrary-length speech insertion and even full sentence generation.

• In the proposed paradigm, global and local factors in speech are explicitly decomposed and separately manipulated to achieve high speaker similarity and continuous prosody.
• Specifically, we proposed to represent the global factors by multiple tokens, which are extracted by cross-attention operation and then injected back by link-attention operation.
• Due to the rich representation of global factors, we manage to achieve high speaker similarity in a zero-shot manner.