2021

Rich Prosody Diversity Modelling with Phone-level Mixture Density Network

Du, Chenpeng, Yu, Kai

Understand

Generating natural speech with diverse and smooth prosody pattern is a challenging task.

Although random sampling with phone-level prosody distribution has been investigated to generate different prosody patterns, the diversity of the generated speech is still very limited and far from what can be achieved by human.
This is largely due to the use of uni-modal distribution, such as single Gaussian, in the prior works of phone-level prosody modelling.
In this work, we propose a novel approach that models phone-level prosodies with GMM based mixture density network (GMM-MDN).

Built on

R. Kubichek, “Mel-cepstral distance measure for objective speech quality assessment,” in Proceedings of IEEE Pacific Rim Conference on Communications Computers and Signal Processing , vol. 1, 1993, pp. 125–128
1993
Earlier work this paper cites.
J. Gauvain and C. Lee, “Maximum a posteriori estimation for multivariate gaussian mixture observations of markov chains,” IEEE Trans. Speech Audio Process. , vol. 2, no. 2, pp. 291–298, 1994
1994
Earlier work this paper cites.
C. M. Bishop, “Mixture density networks,” Aston University , 1994
1994
Earlier work this paper cites.
P. C. Woodland, C. Leggetter, J. Odell, V. Valtchev, and S. Young, “The development of the 1994 htk large vocabulary speech recognition system,” in Proceedings ARPA workshop on spoken language systems technology , 1995, pp. 104–109
1995
Earlier work this paper cites.
M. J. F. Gales, “Maximum likelihood linear transformations for hmm-based speech recognition,” Computer speech & language , vol. 12, no. 2, pp. 75–98, 1998
1998
Earlier work this paper cites.
J. Kominek, T. Schultz, and A. W. Black, “Synthesizer voice quality of new languages calibrated with mean mel cepstral distortion,” in Proc. ISCA SLTU , 2008, pp. 63–68
Original
2008
Earlier work this paper cites.
H. Zen and A. W. Senior, “Deep mixture density networks for acoustic modeling in statistical parametric speech synthesis,” in Proc. IEEE ICASSP , 2014, pp. 3844–3848
2014
Earlier work this paper cites.
V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: An ASR corpus based on public domain audio books,” in Proc. IEEE ICASSP , 2015, pp. 5206–5210
2015
Earlier work this paper cites.
D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in Proc. ICLR , 2015
2015
Earlier work this paper cites.

Similar

Y. Wang, R. J. Skerry-Ryan, D. Stanton et al. , “Tacotron: Towards end-to-end speech synthesis,” in Proc. ISCA Interspeech , 2017, pp. 4006–4010
2017
Cited alongside, same era.
X. Wang, S. Takaki, and J. Yamagishi, “An autoregressive recurrent mixture density network for parametric speech synthesis,” in Proc. IEEE ICASSP , 2017, pp. 4895–4899
2017
Cited alongside, same era.
K. Ito, “The lj speech dataset,” https://keithito.com/LJ-Speech-Dataset/ , 2017
2017
Cited alongside, same era.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in Proc. NIPS , 2017, pp. 5998–6008
2017
Cited alongside, same era.
J. Shen, R. Pang, R. J. Weiss, M. Schuster, N. Jaitly, Z. Yang, Z. Chen, Y. Zhang, Y. Wang, R. Ryan, R. A. Saurous, Y. Agiomyrgiannakis, and Y. Wu, “Natural TTS synthesis by conditioning wavenet on MEL spectrogram predictions,” in Proc. IEEE ICASSP , 2018, pp. 4779–4783
2018
Cited alongside, same era.
N. Li, S. Liu, Y. Liu, S. Zhao, M. Liu, and M. Zhou, “Close to human quality TTS with transformer,” arXiv preprint arXiv:1809.08895 , 2018
Original
2018
Cited alongside, same era.
R. J. Skerry-Ryan, E. Battenberg, Y. Xiao, Y. Wang, D. Stanton, J. Shor, R. J. Weiss, R. Clark, and R. A. Saurous, “Towards end-to-end prosody transfer for expressive speech synthesis with tacotron,” in Proc. ICML , 2018, pp. 4700–4709
2018
Cited alongside, same era.
Y. Wang, D. Stanton, Y. Zhang, R. J. Skerry-Ryan, E. Battenberg, J. Shor, Y. Xiao, Y. Jia, F. Ren, and R. A. Saurous, “Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis,” in Proc. ICML , 2018, pp. 5167–5176
2018
Cited alongside, same era.
K. Akuzawa, Y. Iwasawa, and Y. Matsuo, “Expressive speech synthesis via modeling expressions with variational autoencoder,” in Proc. ISCA Interspeech , 2018, pp. 3067–3071
2018
Cited alongside, same era.

Then

Y. Ren, Y. Ruan, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T. Liu, “Fastspeech: Fast, robust and controllable text to speech,” in Proc. NIPS , 2019, pp. 3165–3174
2019
Later among the works it cites.
Y. Lee and T. Kim, “Robust and fine-grained prosody control of end-to-end speech synthesis,” in Proc. IEEE ICASSP , 2019, pp. 5911–5915
2019
Later among the works it cites.
V. Klimkov, S. Ronanki, J. Rohnke, and T. Drugman, “Fine-grained robust prosody transfer for single-speaker neural text-to-speech,” in Proc. ISCA Interspeech , 2019, pp. 4440–4444
2019
Later among the works it cites.
K. Kumar, R. Kumar, T. de Boissiere, L. Gestin, W. Z. Teoh, J. Sotelo, A. de Brébisson, Y. Bengio, and A. C. Courville, “Melgan: Generative adversarial networks for conditional waveform synthesis,” in Proc. NIPS , 2019, pp. 14 881–14 892
2019
Later among the works it cites.
Y. Ren, C. Hu, T. Qin, S. Zhao, Z. Zhao, and T. Liu, “Fastspeech 2: Fast and high-quality end-to-end text-to-speech,” arXiv preprint arXiv:2006.04558 , 2020
Original
2020
Later among the works it cites.
G. Sun, Y. Zhang, R. J. Weiss, Y. Cao, H. Zen, and Y. Wu, “Fully-hierarchical fine-grained prosody modeling for interpretable speech synthesis,” in Proc. IEEE ICASSP , 2020, pp. 6264–6268
2020
Later among the works it cites.
G. Sun, Y. Zhang, R. J. Weiss, Y. Cao, H. Zen, A. Rosenberg, B. Ramabhadran, and Y. Wu, “Generating diverse and natural text-to-speech samples using a quantized fine-grained VAE and autoregressive prosody prior,” in Proc. IEEE ICASSP , 2020, pp. 6699–6703
2020
Later among the works it cites.
Y. Hono, K. Tsuboi, K. Sawada, K. Hashimoto, K. Oura, Y. Nankaku, and K. Tokuda, “Hierarchical multi-grained generative model for expressive speech synthesis,” in Proc. ISCA Interspeech , 2020, pp. 3441–3445
2020
Later among the works it cites.
C. M. Chien and H. Y. Lee, “Hierarchical prosody modeling for non-autoregressive speech synthesis,” in Proc. IEEE SLT , 2021, pp. 446–453
2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…