2021

Improving Performance of Seen and Unseen Speech Style Transfer in End-to-end Neural TTS

An, Xiaochun, Soong, Frank K., Xie, Lei

Understand

End-to-end neural TTS training has shown improved performance in speech style transfer.

However, the improvement is still limited by the training data in both target styles and speakers.
Inadequate style transfer performance occurs when the trained TTS tries to transfer the speech to a target style from a new speaker with an unknown, arbitrary style.
In this paper, we propose a new approach to style transfer for both seen and unseen styles, with disjoint, multi-style datasets, i.e., datasets of different styles are recorded, each individual style is by one speaker with multiple utterances.

Built on

I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence learning with neural networks,” in Proc. NIPS , 2014, pp. 3104–3112
2014
Earlier work this paper cites.
I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in Proc. NIPS , 2014, pp. 2672–2680
2014
Earlier work this paper cites.
D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” in Proc. ICLR , 2015
2015
Earlier work this paper cites.
D. P. Kingma, T. Salimans, R. Jozefowicz, X. Chen, I. Sutskever, and M. Welling, “Improving variational inference with inverse autoregressive flow,” in Proc. NIPS , 2016, pp. 4743–4751
2016
Earlier work this paper cites.
M. Mathieu, J. Zhao, P. Sprechmann, A. Ramesh, and Y. LeCun, “Disentangling factors of variation in deep representations using adversarial training,” in Proc. NIPS , 2016
2016
Earlier work this paper cites.
Y. Wang, R. Skerry-Ryan, D. Stanton, Y. Wu, R. J. Weiss, N. Jaitly, Z. Yang, Y. Xiao, Z. Chen, S. Bengio, Q. Le, Y. Agiomyrgiannakis, R. Clark, and R. A. Saurous, “Tacotron: Towards end-to-end speech synthesis,” in Proc. INTERSPEECH , 2017, pp. 4006–4010
2017
Earlier work this paper cites.
J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networks,” in Proc. ICCV , 2017, pp. 2223–2232
2017
Earlier work this paper cites.
W. Ping, K. Peng, A. Gibiansky, S. O. Arik, A. Kannan, S. Narang, J. Raiman, and J. J. Miller, “Deep voice 3: 2000-speaker neural text-to-speech,” in Proc. ICLR , 2018
2018
Earlier work this paper cites.
J. Shen, R. Pang, R. J. Weiss, M. Schuster, N. Jaitly, Z. Yang, Z. Chen, Y. Zhang, Y. Wang, R. Skerrv-Ryan, R. A. Saurous, Y. Agiomyrgiannakis, and Y. Wu, “Natural TTS synthesis by conditioning WaveNet on mel spectrogram predictions,” in Proc. ICASSP , 2018, pp. 4779–4783
2018
Earlier work this paper cites.
Y. Jia, Y. Zhang, R. J. Weiss, Q. Wang, J. Shen, F. Ren, Z. Chen, P. Nguyen, R. Pang, I. L. Moreno, and Y. Wu, “Transfer learning from speaker verification to multispeaker text-to-speech synthesis,” in Proc. NIPS , 2018, pp. 4485–4495
2018
Earlier work this paper cites.
D. Stanton, Y. Wang, and R. Skerry-Ryan, “Predicting expressive speaking style from text in end-to-end speech synthesis,” in Proc. SLT , 2018, pp. 595–602
2018
Earlier work this paper cites.

Similar

R. Skerry-Ryan, E. Battenberg, Y. Xiao, Y. Wang, D. Stanton, J. Shor, R. J. Weiss, R. Clark, and R. A. Saurous, “Towards end-to-end prosody transfer for expressive speech synthesis with Tacotron,” in Proc. ICML , 2018, pp. 4693–4702
2018
Cited alongside, same era.
K. Akuzawa, Y. Iwasawa, and Y. Matsuo, “Expressive speech synthesis via modeling expressions with variational autoencoder,” in Proc. INTERSPEECH , 2018, pp. 3067–3071
2018
Cited alongside, same era.
Y. Wang, D. Stanton, Y. Zhang, R. Skerry-Ryan, E. Battenberg, J. Shor, Y. Xiao, F. Ren, Y. Jia, and R. A. Saurous, “Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis,” in Proc. ICML , 2018, pp. 5180–5189
2018
Cited alongside, same era.
F. Yang, S. Yang, P. Zhu, P. Yan, and L. Xie, “Improving mandarin end-to-end speech synthesis by self-attention and learnable gaussian bias,” in Proc. ASRU , 2019, pp. 208–213
2019
Cited alongside, same era.
P. Wu, Z. Ling, L. Liu, Y. Jiang, H. Wu, and L. Dai, “End-to-end emotional speech synthesis using style tokens and semi-supervised training,” in Proc. APSIPA ASC , 2019
2019
Cited alongside, same era.
X. An, Y. Wang, S. Yang, Z. Ma, and L. Xie, “Learning hierarchical representations for expressive speaking style in end-to-end speech synthesis,” in Proc. ASRU , 2019, pp. 184–191
2019
Cited alongside, same era.
W. Hsu, Y. Zhang, R. J. Weiss, H. Zen, Y. Wu, Y. Wang, Y. Cao, Y. Jia, Z. Chen, J. Shen, P. Nguyen, and R. Pang, “Hierarchical generative modeling for controllable speech synthesis,” in Proc. ICLR , 2019
2019
Cited alongside, same era.
Y. Zhang, S. Pan, L. He, and Z. Ling, “Learning latent representations for style control and transfer in end-to-end speech synthesis,” in Proc. ICASSP , 2019, pp. 6945–6949
2019
Cited alongside, same era.
Y. Bian, C. Chen, Y. Kang, and Z. Pan, “Multi-reference tacotron by intercross training for style disentangling, transfer and control in speech synthesis,” in Proc. INTERSPEECH , 2019
2019
Cited alongside, same era.
J. Valin and J. Skoglund, “LPCNET: Improving neural speech synthesis through linear prediction,” in Proc. ICASSP , 2019, pp. 5891–5895
2019
Cited alongside, same era.

Then

H. Guo, F. K. Soong, L. He, and L. Xie, “A new GAN-based end-to-end TTS training algorithm,” in Proc. INTERSPEECH , 2019
2019
Later among the works it cites.
G. Sun, Y. Zhang, R. J. Weiss, Y. Cao, H. Zen, A. Rosenberg, B. Ramabhadran, and Y. Wu, “Generating diverse and natural text-to-speech samples using a quantized fine-grained VAE and autoregressive prosody prior,” in Proc. ICASSP , 2020
2020
Later among the works it cites.
R. Liu, B. Sisman, G. Gao, and H. Li, “Expressive TTS training with frame and style reconstruction loss,” arXiv preprint arXiv:2008.01490 , 2020
Original
2020
Later among the works it cites.
R. Habib, S. Mariooryad, M. Shannon, E. Battenberg, R. Skerry-Ryan, D. Stanton, D. T. H. Kao, and T. Bagby, “Semi-supervised generative modeling for controllable speech synthesis,” in Proc. ICLR , 2020
2020
Later among the works it cites.
G. Sun, Y. Zhang, R. J. Weiss, Y. Cao, H. Zen, and Y. Wu, “Fully-hierarchical fine-grained prosody modelling for interpretable speech synthesis,” in Proc. ICASSP , 2020
2020
Later among the works it cites.
M. Whitehill, S. Ma, D. McDuff, and Y. Song, “Multi-reference neural TTS stylization with adversarial cycle consistency,” in Proc. INTERSPEECH , 2020, pp. 4442–4446
2020
Later among the works it cites.
V. Aggarwal, M. Cotescu, N. Prateek, J. Lorenzo-Trueba, and R. Barra-Chicote, “Using VAEs and normalizing flows for one-shot text-to-speech synthesis of expressive speech,” in Proc. ICASSP , 2020, pp. 6179–6183
Original
2020
Later among the works it cites.
Y. Lei, S. Yang, and L. Xie, “Fine-grained emotion strength transfer, control and prediction for emotional speech synthesis,” in Proc. SLT , 2021
2021
Closest in time.
T. Li, S. Yang, L. Xue, and L. Xie, “Controllable emotion transfer for end-to-end speech synthesis,” in Proc. ISCSLP , 2021, pp. 1–5
2021
Closest in time.
L. Xue, S. Pan, L. He, L. Xie, and F. K. Soong, “Cycle consistent network for end-to-end style transfer TTS training,” Neural Networks , vol. 140, pp. 223–236, 2021
2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…