Learning Robust Latent Representations for Controllable Speech Synthesis

Understand

State-of-the-art Variational Auto-Encoders (VAEs) for learning disentangled latent representations give impressive results in discovering features like pitch, pause duration, and accent in speech data, leading to highly controllable text-to-speech (TTS) synthesis.

• However, these LSTM-based VAEs fail to learn latent clusters of speaker attributes when trained on either limited or noisy datasets.
• Further, different latent variables start encoding the same features, limiting the control and expressiveness during speech synthesis.
• To resolve these issues, we propose RTI-VAE (Reordered Transformer with Information reduction VAE) where we minimize the mutual information between different latent variables and devise a modified Transformer architecture with layer reordering to learn controllable latent representations in speech data.