2020

Improving Prosody Modelling with Cross-Utterance BERT Embeddings for End-to-end Speech Synthesis

Xu, Guanghui, Song, Wei, Zhang, Zhengchen et al.

Understand

Despite prosody is related to the linguistic information up to the discourse structure, most text-to-speech (TTS) systems only take into account that within each sentence, which makes it challenging when converting a paragraph of texts into natural and expressive speech.

In this paper, we propose to use the text embeddings of the neighboring sentences to improve the prosody generation for each utterance of a paragraph in an end-to-end fashion without using any explicit prosody features.
More specifically, cross-utterance (CU) context vectors, which are produced by an additional CU encoder based on the sentence embeddings extracted by a pre-trained BERT model, are used to augment the input of the Tacotron2 decoder.
Two types of BERT embeddings are investigated, which leads to the use of different CU encoder structures.

Built on

F. Eyben, S. Buchholz, N. Braunschweiler, J. Latorre, V. Wan, M. Gales, and K. Knill, “Unsupervised clustering of emotion and voice styles for expressive TTS,” in
2012
Earlier work this paper cites.
H. Zen, A. Senior, and M. Schuster, “Statistical parametric speech synthesis using deep neural networks,” in
2013
Earlier work this paper cites.
L. Chen, M. Gales, N. Braunschweiler, M. Akamine, and K. Knill, “Integrated expression prediction and speech synthesis from text,”
2013
Earlier work this paper cites.
F. Tesser, G. Sommavilla, G. Paci, and P. Cosi, “Experiments with signal-driven symbolic prosody for statistical parametric speech synthesis,” in
2013
Earlier work this paper cites.
Y. Fan, Y. Qian, F.-L. Xie, and F. Soong, “TTS synthesis with bidirectional LSTM based recurrent neural networks,” in
2014
Earlier work this paper cites.
D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” in
2014
Earlier work this paper cites.
I. Sutskever, O. Vinyals, and Q. Le, “Sequence to sequence learning with neural networks,” in
2014
Earlier work this paper cites.
D. Kingma and M. Welling, “Auto-encoding variational Bayes,” in
2014
Earlier work this paper cites.
O. Watts, Z. Wu, and S. King, “Sentence-level control vectors for deep neural network speech synthesis,” in
2015
Earlier work this paper cites.
H. Liu, H. Lu, X. Shao, and Y. Xu, “Model-based parametric prosody synthesis with deep neural network,” in
2016
Earlier work this paper cites.
N. Hu, P. Shao, Y. Zu, Z. Wang, W. Huang, and S. Wang, “Discourse prosody and its application to speech synthesis,” in
2016
Earlier work this paper cites.
M. Farrús, C. Lai, and J. Moore, “Paragraph-based prosodic cues for speech synthesis applications,” in
2016
Earlier work this paper cites.
J. Sotelo, S. Mehri, K. Kumar, J. F. Santos, K. Kastner, A. C. Courville, and Y. Bengio, “Char2wav: End-to-end speech synthesis,” in
2017
Earlier work this paper cites.

Similar

Y. Wang, R. Skerry-Ryan, D. Stanton, Y. Wu, R. Weiss, N. Jaitly, Z. Yang, Y. Xiao, Z. Chen, S. Bengio, Q. Le, Y. Agiomyrgiannakis, R. Clark, and R. Saurous, “Tacotron: Towards end-to-end speech synthesis,” in
2017
Cited alongside, same era.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in
2017
Cited alongside, same era.
K. Ito and L. Johnson, “The LJ speech dataset,”
2017
Cited alongside, same era.
J. Shen, R. Pang, R. J. Weiss, M. Schuster, N. Jaitly, Z. Yang, Z. Chen, Y. Zhang, Y. Wang, R. Ryan, R. Saurous, Y. Agiomyrgiannakis, and Y. Wu, “Natural TTS synthesis by conditioning WaveNet on Mel spectrogram predictions,” in
2018
Cited alongside, same era.
W. Ping, K. Peng, A. Gibiansky, S. Arik, A. Kannan, S. Narang, J. Raiman, and J. Miller, “Deep Voice 3: Scaling text-to-speech with convolutional sequence learning,” in
2018
Cited alongside, same era.
Y. Taigman, L. Wolf, A. Polyak, and E. Nachmani, “Voiceloop: Voice fitting and synthesis via a phonological loop,” in
2018
Cited alongside, same era.
R. Skerry-Ryan, E. Battenberg, Y. Xiao, Y. Wang, D. Stanton, J. Shor, R. Weiss, R. Clark, and R. A. Saurous, “Towards end-to-end prosody transfer for expressive speech synthesis with Tacotron,” in
2018
Cited alongside, same era.
X. Wu, Y. Cao, M. Wang, S. Liu, S. Kang, Z. Wu, X. Liu, D. Su, D. Yu, and H. Meng, “Rapid style adaptation using residual error embedding for expressive speech synthesis.” in
2018
Cited alongside, same era.
Y. Wang, D. Stanton, Y. Zhang, R.-S. Ryan, E. Battenberg, J. Shor, Y. Xiao, Y. Jia, F. Ren, and R. A. Saurous, “Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis,” in
2018
Cited alongside, same era.
G. E. Henter, J. Lorenzo-Trueba, X. Wang, and J. Yamagishi, “Deep encoder-decoder models for unsupervised learning of controllable speech synthesis,” in
Original
2018
Cited alongside, same era.
K. Akuzawa, Y. Iwasawa, and Y. Matsuo, “Expressive speech synthesis via modeling expressions with variational autoencoder,” in
2018
Cited alongside, same era.
X. Zhu, L. Xie, X. Chen, X. Lou, X. Zhu, and X. Tan, “Controlling expressivity using input codes in neural network based TTS,” in
2018
Cited alongside, same era.

Then

N. Kalchbrenner, E. Elsen, K. Simonyan, S. Noury, N. Casagrande, E. Lockhart, F. Stimberg, A. van den Oord, S. Dieleman, and K. Kavukcuoglu, “Efficient neural audio synthesis,” in
2018
Later among the works it cites.
M. Schoeffler, S. Bartoschek, F.-R. Stöter, M. Roess, S. Westphal, B. Edler, and J. Herre, “webMUSHRA – A comprehensive framework for web-based listening tests,”
2018
Later among the works it cites.
Y. Lee and T. Kim, “Robust and fine-grained prosody control of end-to-end speech synthesis,” in
2019
Later among the works it cites.
Y.-J. Zhang, S. Pan, L. He, and Z.-H. Ling, “Learning latent representations for style control and transfer in end-to-end speech synthesis,” in
2019
Later among the works it cites.
W. Hsu, Y. Zhang, R. J. Weiss, H. Zen, Y. Wu, Y. Wang, Y. Cao, Y. Jia, Z. Chen, J. Shen, P. Nguyen, and R. Pang, “Hierarchical generative modeling for controllable speech synthesis,” in
2019
Later among the works it cites.
J. Devlin, M. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional Transformers for language understanding,” in
2019
Later among the works it cites.
T. Hayashi, S. Watanabe, T. Toda, K. Takeda, S. Toshniwal, and K. Livescu, “Pre-trained text embeddings for enhanced text-to-speech synthesis,” in
2019
Later among the works it cites.
W. Fang, Y. Chung, and J. Glass, “Towards transfer learning for end-to-end speech synthesis from deep pre-trained language models,” in
Original
2019
Later among the works it cites.
A. Aubin, A. Cervone, O. Watts, and S. King, “Improving speech synthesis with discourse relations,” in
2019
Later among the works it cites.
T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, and J. Brew, “HuggingFace’s Transformers: State-of-the-art natural language processing,” in
Original
2019
Later among the works it cites.
G. Sun, Y. Zhang, R. J. Weiss, Y. Cao, H. Zen, A. Rosenberg, B. Ramabhadran, and Y. Wu, “Generating diverse and natural text-to-speech samples using a quantized fine-grained VAE and autoregressive prosody prior,” in
2020
Closest in time.
G. Sun, Y. Zhang, R. J. Weiss, Y. Cao, H. Zen, and Y. Wu, “Fully-hierarchical fine-grained prosody modeling for interpretable speech synthesis,” in
2020
Closest in time.
Y. Xiao, L. He, H. Ming, and F. Soong, “Improving prosody with linguistic and BERT derived features in multi-speaker based mandarin chinese neural TTS,” in
2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…