2020

Flowtron: an Autoregressive Flow-based Generative Network for Text-to-Speech Synthesis

Valle, Rafael, Shih, Kevin, Prenger, Ryan et al.

Understand

In this paper we propose Flowtron: an autoregressive flow-based generative network for text-to-speech synthesis with control over speech variation and style transfer.

Flowtron borrows insights from IAF and revamps Tacotron in order to provide high-quality and expressive mel-spectrogram synthesis.
Flowtron is optimized by maximizing the likelihood of the training data, which makes training simple and stable.
Flowtron learns an invertible mapping of data to a latent space that can be manipulated to control many aspects of speech synthesis (pitch, tone, speech rate, cadence, accent).

Built on

Mellotron: Multispeaker expressive voice synthesis by conditioning on rhythm, pitch and global style tokens
Original
Valle, R., Li, J., Prenger, R., and Catanzaro, B · 1910
Earlier work this paper cites.
Synthesis of fairy tales using an analog vocal tract
Umeda, N., Matsui, E., Suzuki, T., and Omura, H · 1968
Earlier work this paper cites.
Wargames, 1983
Badham, J., Lasker, L., Parkes, W. F., Rubinstein, A. B., Broderick, M., Coleman, D., and Wood, J · 1983
Earlier work this paper cites.
The cmu pronouncing dictionary
Weide, R. L · 1998
Earlier work this paper cites.
Yin, a fundamental frequency estimator for speech and music
De Cheveigné, A. and Kawahara, H · 2002
Earlier work this paper cites.
Nice: Non-linear independent components estimation
Original
Dinh, L., Krueger, D., and Bengio, Y · 2014
Earlier work this paper cites.
Adam: A method for stochastic optimization
Original
Kingma, D. P. and Ba, J · 2014
Earlier work this paper cites.
Unsupervised representation learning with deep convolutional generative adversarial networks
Original
Radford, A., Metz, L., and Chintala, S · 2015
Earlier work this paper cites.
Grammar as a foreign language
Vinyals, O., Kaiser, Ł., Koo, T., Petrov, S., Sutskever, I., and Hinton, G · 2015
Earlier work this paper cites.
Density estimation using real nvp
Original
Dinh, L., Sohl-Dickstein, J., and Bengio, S · 2016
Earlier work this paper cites.

Similar

Improved variational inference with inverse autoregressive flow
Kingma, D. P., Salimans, T., Jozefowicz, R., Chen, X., Sutskever, I., and Welling, M · 2016
Cited alongside, same era.
Singing voice synthesis based on deep neural networks
Nishimura, M., Hashimoto, K., Oura, K., Nankaku, Y., and Tokuda, K · 2016
Cited alongside, same era.
Deep voice 2: Multi-speaker neural text-to-speech
Gibiansky, A., Arik, S., Diamos, G., Miller, J., Peng, K., Ping, W., Raiman, J., and Zhou, Y · 2017
Cited alongside, same era.
The LJ speech dataset, 2017
Ito, K. et al · 2017
Cited alongside, same era.
Deep voice 3: 2000-speaker neural text-to-speech
Original
Ping, W., Peng, K., Gibiansky, A., Arik, S. O., Kannan, A., Narang, S., Raiman, J., and Miller, J · 2017
Cited alongside, same era.
Natural tts synthesis by conditioning wavenet on mel spectrogram predictions
Original
Shen, J., Pang, R., Weiss, R. J., Schuster, M., Jaitly, N., Yang, Z., Chen, Z., Zhang, Y., Wang, Y., Skerry-Ryan, R., et al · 2017
Cited alongside, same era.
Tacotron: A fully end-to-end text-to-speech synthesis model
Original
Wang, Y., Skerry-Ryan, R., Stanton, D., Wu, Y., Weiss, R. J., Jaitly, N., Yang, Z., Xiao, Y., Chen, Z., Bengio, S., et al · 2017
Cited alongside, same era.
Expressive speech synthesis via modeling expressions with variational autoencoder
Original
Akuzawa, K., Iwasawa, Y., and Matsuo, Y · 2018
Cited alongside, same era.
Hierarchical generative modeling for controllable speech synthesis
Original
Hsu, W.-N., Zhang, Y., Weiss, R. J., Zen, H., Wu, Y., Wang, Y., Cao, Y., Jia, Y., Chen, Z., Shen, J., et al · 2018
Cited alongside, same era.
Deep voice 2: Multi-speaker neural text-to-speech
Original
Arik, S., Diamos, G., Gibiansky, A., Miller, J., Peng, K., Ping, W., Raiman, J., and Zhou, Y
Cited in the paper.
Deep voice: Real-time neural text-to-speech
Original
Arik, S. O., Chrzanowski, M., Coates, A., Diamos, G., Gibiansky, A., Kang, Y., Li, X., Miller, J., Ng, A., Raiman, J., et al
Cited in the paper.
Mellotron github repo, 2019a
Valle, R., Li, J., Prenger, R., and Catanzaro, B
Cited in the paper.

Then

Glow: Generative flow with invertible 1x1 convolutions
Original
Kingma, D. P. and Dhariwal, P · 2018
Later among the works it cites.
Image transformer
Original
Parmar, N., Vaswani, A., Uszkoreit, J., Kaiser, Ł., Shazeer, N., Ku, A., and Tran, D · 2018
Later among the works it cites.
Towards end-to-end prosody transfer for expressive speech synthesis with tacotron
Original
Skerry-Ryan, R., Battenberg, E., Xiao, Y., Wang, Y., Stanton, D., Shor, J., Weiss, R. J., Clark, R., and Saurous, R. A · 2018
Later among the works it cites.
Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis
Original
Wang, Y., Stanton, D., Zhang, Y., Skerry-Ryan, R., Battenberg, E., Shor, J., Xiao, Y., Ren, F., Jia, Y., and Saurous, R. A · 2018
Later among the works it cites.
High fidelity speech synthesis with adversarial networks
Original
Bińkowski, M., Donahue, J., Dieleman, S., Clark, A., Elsen, E., Casagrande, N., Cobo, L. C., and Simonyan, K · 2019
Later among the works it cites.
Transflow learning: Repurposing flow models without retraining
Original
Gambardella, A., Baydin, A. G., and Torr, P. H · 2019
Later among the works it cites.
Adversarially trained end-to-end korean singing voice synthesis system
Original
Lee, J., Choi, H.-S., Jeon, C.-B., Koo, J., and Lee, K · 2019
Later among the works it cites.
Waveglow: A flow-based generative network for speech synthesis
Prenger, R., Valle, R., and Catanzaro, B · 2019
Later among the works it cites.
Libritts: A corpus derived from librispeech for text-to-speech
Original
Zen, H., Dang, V., Clark, R., Zhang, Y., Weiss, R. J., Jia, Y., Chen, Z., and Wu, Y · 2019
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…