A Multi-Stage Multi-Codebook VQ-VAE Approach to High-Performance Neural TTS

Understand

We propose a Multi-Stage, Multi-Codebook (MSMC) approach to high-performance neural TTS synthesis.

• A vector-quantized, variational autoencoder (VQ-VAE) based feature analyzer is used to encode Mel spectrograms of speech training data by down-sampling progressively in multiple stages into MSMC Representations (MSMCRs) with different time resolutions, and quantizing them with multiple VQ codebooks, respectively.
• Multi-stage predictors are trained to map the input text sequence to MSMCRs progressively by minimizing a combined loss of the reconstruction Mean Square Error (MSE) and "triplet loss".
• In synthesis, the neural vocoder converts the predicted MSMCRs into final speech waveforms.