2021

Alternate Endings: Improving Prosody for Incremental Neural TTS with Predicted Future Text Input

Stephenson, Brooke, Hueber, Thomas, Girin, Laurent et al.

Understand

The prosody of a spoken word is determined by its surrounding context.

In incremental text-to-speech synthesis, where the synthesizer produces an output before it has access to the complete input, the full context is often unknown which can result in a loss of naturalness in the synthesized speech.
In this paper, we investigate whether the use of predicted future text can attenuate this loss.
We compare several test conditions of next future word: (a) unknown (zero-word), (b) language model predicted, (c) randomly predicted and (d) ground-truth.

Built on

J. ’t Hart, “Differential sensitivity to pitch distance, particularly in speech,” The Journal of the Acoustical Society of America , vol. 69, no. 3, pp. 811–821, 1981
1981
Earlier work this paper cites.
H. Quené, “On the just noticeable difference for tempo in speech,” Journal of Phonetics , vol. 35, no. 3, pp. 353–362, 2007
2007
Earlier work this paper cites.
T. Baumann and D. Schlangen, “Evaluating prosodic processing for incremental speech synthesis,” in Proc. of Interspeech , Portland, OR, USA, 2012, pp. 438–441
2012
Earlier work this paper cites.
T. Baumann, “Partial representations improve the prosody of incremental speech synthesis,” in Proc. of Interspeech , Singapore, 2014, pp. 2932–2936
2014
Earlier work this paper cites.
T. Baumann, “Decision tree usage for incremental parametric speech synthesis,” in Proc. of ICASSP , Florence, Italy, 2014, pp. 3819–3823
2014
Earlier work this paper cites.
M. Pouget, T. Hueber, G. Bailly, and T. Baumann, “HMM training strategy for incremental speech synthesis,” in Proc. of Interspeech , Dresden, Germany, 2015, pp. 1201–1205
2015
Earlier work this paper cites.
Brian McFee, Colin Raffel, Dawen Liang, Daniel P.W. Ellis, Matt McVicar, Eric Battenberg, and Oriol Nieto, “Librosa: Audio and music signal analysis in Python,” in Proc. of the Python in Science Conference , 2015, pp. 18–24
2015
Earlier work this paper cites.
M. Pouget, O. Nahorna, T. Hueber, and G. Bailly, “Adaptive latency for part-of-speech tagging in incremental text-to-speech synthesis,” in Proc. of Interspeech , San Francisco, CA, USA, 2016, pp. 2846–2850
2016
Earlier work this paper cites.
S. G. Luke and K. Christianson, “Limits on lexical prediction during reading,” Cognitive Psychology , vol. 88, pp. 22–60, 2016
2016
Earlier work this paper cites.
K. Ito, “The LJ speech dataset,” https://keithito.com/LJ-Speech-Dataset/, 2017
2017
Earlier work this paper cites.
R. Speer, J. Chin, A. Lin, S. Jewett, and L. Nathan, “Luminosoinsight/wordfreq: v2.2 [Computer software],” Oct. 2018. [Online]. Available: https://doi.org/10.5281/zenodo.1443582
2018
Earlier work this paper cites.

Similar

P. Boersma and D. Weenink, “Praat: doing phonetics by computer [Computer software],” Version 6.0.37, retrieved 3 February 2018 http://www.praat.org/, 2018
2018
Cited alongside, same era.
Y. Jadoul, B. Thompson, and B. de Boer, “Introducing Parselmouth: A Python interface to Praat,” Journal of Phonetics , vol. 71, pp. 1–15, 2018
2018
Cited alongside, same era.
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in Proc. of ACL , Florence, Italy, 2019, pp. 4171–4186
2019
Cited alongside, same era.
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Language models are unsupervised multitask learners,” OpenAI blog , vol. 1, no. 8, 2019
2019
Cited alongside, same era.
T. Yanagita, S. Sakti, and S. Nakamura, “Neural iTTS: Toward synthesizing speech in real-time with end-to-end neural text-to-speech framework,” in Proc. of SSW , Vienna, Austria, 2019, pp. 183–188
2019
Cited alongside, same era.
W. Fang, Y.-A. Chung, and J. Glass, “Towards transfer learning for end-to-end speech synthesis from deep pre-trained language models,” arXiv preprint arXiv:1906.07307 , 2019
Original
2019
Cited alongside, same era.
Y.-A. Chung, Y. Wang, W.-N. Hsu, Y. Zhang, and R. Skerry-Ryan, “Semi-supervised training for improving data efficiency in end-to-end speech synthesis,” in Proc. of ICASSP , Brighton, United Kingdom, 2019, pp. 6940–6944
2019
Cited alongside, same era.
R. Zheng, M. Ma, B. Zheng, and L. Huang, “Speculative beam search for simultaneous translation,” in Proc. of EMNLP-IJCNLP , Hong Kong, China, 2019, pp. 1395–1402
2019
Cited alongside, same era.
H. Zen, V. Dang, R. Clark, Y. Zhang, R. J. Weiss, Y. Jia, Z. Chen, and Y. Wu, “LibriTTS: A corpus derived from LibriSpeech for text-to-speech,” in Proc. of Interspeech , Graz, Austria, 2019, pp. 1526–1530
2019
Cited alongside, same era.
E. Delasalles, S. Lamprier, and L. Denoyer, “Learning Dynamic Author Representations with Temporal Language Models,” in 2019 IEEE International Conference on Data Mining (ICDM) . Beijing, China: IEEE, Nov. 2019, pp. 120–129
2019
Cited alongside, same era.
B. Stephenson, L. Besacier, L. Girin, and T. Hueber, “What the future brings: Investigating the impact of lookahead for incremental neural TTS,” in Proc. of Interspeech , Shanghai, China, 2020, pp. 215–219
2020
Cited alongside, same era.
ITU-R, “Recommandation BS.1534 and BS.1116: Methods for the subjective assessment of small impairments in audio systems.”
Cited in the paper.

Then

M. Ma, B. Zheng, K. Liu, R. Zheng, H. Liu, K. Peng, K. Church, and L. Huang, “Incremental text-to-speech synthesis with prefix-to-prefix framework,” in Findings of ACL: EMNLP 2020 , 2020, pp. 3886–3896
2020
Later among the works it cites.
D. Mohan, R. Lenain, L. Foglianti, T. H. Teh, M. Staib, A. Torresquintero, and J. Gao, “Incremental text-to-speech for neural sequence-to-sequence models using reinforcement learning,” in Proc. of Interspeech , Shanghai, China, 2020, pp. 3186–3190
2020
Later among the works it cites.
K. Sudoh, T. Kano, S. Novitasari, T. Yanagita, S. Sakti, and S. Nakamura, “Simultaneous speech-to-speech translation system with neural incremental asr, mt, and tts,” arXiv preprint arXiv:2011.04845 , 2020
Original
2020
Later among the works it cites.
S. Novitasari, A. Tjandra, T. Yanagita, S. Sakti, and S. Nakamura, “Incremental machine speech chain towards enabling listening while speaking in real-time,” arXiv preprint arXiv:2011.02126 , 2020
Original
2020
Later among the works it cites.
T. Saeki, S. Takamichi, and H. Saruwatari, “Incremental text-to-speech synthesis using pseudo lookahead with large pretrained language model,” arXiv preprint arXiv:2012.12612 , 2020
Original
2020
Later among the works it cites.
Y. Ren, C. Hu, T. Qin, S. Zhao, Z. Zhao, and T.-Y. Liu, “Fastspeech 2: Fast and high-quality end-to-end text-to-speech,” arXiv preprint arXiv:2006.04558 , 2020
Original
2020
Later among the works it cites.
T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, and colleagues, “Transformers: State-of-the-art natural language processing,” in Proc. of ACL EMNLP , Online, 2020, pp. 38–45
2020
Later among the works it cites.
T. Hayashi, R. Yamamoto, K. Inoue, T. Yoshimura, S. Watanabe, T. Toda, K. Takeda, Y. Zhang, and X. Tan, “Espnet-TTS: Unified, reproducible, and integratable open source end-to-end text-to-speech toolkit,” in Proc. of ICASSP , Barcelona, Spain, 2020, pp. 7654–7658
2020
Later among the works it cites.
R. Yamamoto, E. Song, and J.-M. Kim, “Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram,” in Proc. of ICASSP , Barcelona, Spain, 2020, pp. 6199–6203
2020
Later among the works it cites.
T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, and colleagues, “Language models are few-shot learners,” arXiv preprint arXiv:2005.14165 , 2020
Original
2020
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…