2016

Fast, Compact, and High Quality LSTM-RNN Based Statistical Parametric Speech Synthesizers for Mobile Devices

Zen, Heiga, Agiomyrgiannakis, Yannis, Egberts, Niels et al.

Understand

Acoustic models based on long short-term memory recurrent neural networks (LSTM-RNNs) were applied to statistical parametric speech synthesis (SPSS) and showed significant improvements in naturalness and latency over those based on hidden Markov models (HMMs).

This paper describes further optimizations of LSTM-RNN-based SPSS for deployment on mobile devices; weight quantization, multi-frame inference, and robust inference using an {\epsilon}-contaminated Gaussian loss function.
Experimental results in subjective listening tests show that these optimizations can make LSTM-RNN-based SPSS comparable to HMM-based SPSS in runtime speed while maintaining naturalness.
Evaluations between LSTM-RNN- based SPSS and HMM-driven unit selection speech synthesis are also presented.

Built on

“A survey of sampling from contaminated distributions,”
J. Tukey, · 1960
Earlier work this paper cites.
“TTS synthesis with bidirectional LSTM based recurrent neural networks,”
Y. Fan, Y. Qian, and F. Soong, · 1968
Earlier work this paper cites.
“Modelling of continuous speech observations,”
A. Richter, · 1986
Earlier work this paper cites.
The acoustic modeling problem in automatic speech recognition
P. Brown, · 1987
Earlier work this paper cites.
“Static and dynamic error propagation networks with application to speech coding,”
A. Robinson and F. Fallside, · 1988
Earlier work this paper cites.
“Backpropagation through time: what it does and how to do it,”
P. Werbos, · 1990
Earlier work this paper cites.
“Long short-term memory,”
S. Hochreiter and J. Schmidhuber, · 1997
Earlier work this paper cites.
“Vector quantization of speech spectral parameters using statistics of dynamic features,”
K. Koishida, K. Tokuda, T. Masuko, and T. Kobayashi, · 1997
Earlier work this paper cites.
“Tail distribution modelling using the Richter and power exponential distributions,”
M. Gales and P. Olsen, · 1999
Earlier work this paper cites.
“Speech parameter generation algorithms for HMM-based speech synthesis,”
K. Tokuda, T. Yoshimura, T. Masuko, T. Kobayashi, and T. Kitamura, · 2000
Earlier work this paper cites.
Simultaneous modeling of phonetic and prosodic parameters, and characteristic conversion for HMM-based text-to-speech systems
T. Yoshimura, · 2002
Earlier work this paper cites.
“The hidden Markov model toolkit (HTK) version 3.4,” http://htk.eng.cam.ac.uk/
S. Young, G. Evermann, M. Gales, T. Hain, D. Kershaw, X.-Y. Liu, G. Moore, J. Odell, D. Ollason, D. Povey, V. Valtchev, and P. Woodland, · 2006
Earlier work this paper cites.
“NEON technology introduction,”
V. Reddy, · 2008
Earlier work this paper cites.
“Statistical parametric speech synthesis,”
H. Zen, K. Tokuda, and A. Black, · 2009
Earlier work this paper cites.
“Probablistic modelling of F0 in unvoiced regions in HMM based speech synthesis,”
K. Yu, T. Toda, M. Găsíc, S. Keizer, F. Mairesse, B. Thomson, and S. Young, · 2009
Earlier work this paper cites.
“Quantized HMMs for low footprint text-to-speech synthesis,”
A. Gutkin, J. Gonzalvo, S. Breuer, and P. Taylor, · 2010
Earlier work this paper cites.

Similar

“Statistical parametric speech synthesis using deep neural networks,”
H. Zen, A. Senior, and M. Schuster, · 2013
Cited alongside, same era.
“Combining a vector space representation of linguistic context with a deep neural network for text-to-speech synthesis,”
H. Lu, S. King, and O. Watts, · 2013
Cited alongside, same era.
“Deep learning in speech synthesis,” http://research.google.com/pubs/archive/41539.pdf
H. Zen, · 2013
Cited alongside, same era.
“Multiframe deep neural networks for acoustic modeling,”
V. Vanhoucke, M. Devin, and G. Heigold, · 2013
Cited alongside, same era.
“Loss minimization and parameter estimation with heavy tails,”
Original
D. Hsu and S. Sabato, · 2013
Cited alongside, same era.
“On rectified linear units for speech processing,”
M. Zeiler, M. Ranzato, R. Monga, M. Mao, K. Yang, Q.-V. Le, P. Nguyen, A. Senior, V. Vanhoucke, J. Dean, and G. Hinton, · 2013
Cited alongside, same era.
“An empirical study of learning rates in deep neural networks for speech recognition,”
A. Senior, G. Heigold, M. Ranzato, and K. Yang, · 2013
Cited alongside, same era.
“On the training aspects of deep neural network (DNN) for parametric TTS synthesis,”
Y. Qian, Y. Fan, W. Hu, and F. Soong, · 2014
Cited alongside, same era.
“Voice source modelling using deep neural networks for statistical parametric speech synthesis,”
T. Raitio, H. Lu, J. Kane, A. Suni, M. Vainio, S. King, and P. Alku, · 2014
Cited alongside, same era.
“Modeling DCT parameterized F0 trajectory at intonation phrase level with DNN or decision tree,”
X. Yin, M. Lei, Y. Qian, F. Soong, L. He, Z.-H. Ling, and L.-R. Dai, · 2014
Cited alongside, same era.
“Deep mixture density networks for acoustic modeling in statistical parametric speech synthesis,”
H. Zen and A. Senior, · 2014
Cited alongside, same era.
“Prosody contour prediction with long short-term memory, bi-directional, deep recurrent neural networks,”
R. Fernandez, A. Rendel, B. Ramabhadran, and R. Hoory, · 2014
Cited alongside, same era.

Then

“Unidirectional long short-term memory recurrent neural network with recurrent output layer for low-latency speech synthesis,”
H. Zen and H. Sak, · 2015
Later among the works it cites.
“Fusion of multiple parameterisations for DNN-based sinusoidal speech synthesis with multi-task learning,”
Q. Hu, Z. Wu, K. Richmond, J. Yamagishi, Y. Stylianou, and R. Maia, · 2015
Later among the works it cites.
“Minimum trajectory error training for deep neural networks, combined with stacked bottleneck features,”
Z. Wu and S. King, · 2015
Later among the works it cites.
“Deep neural networks employing multi-task learning and stacked bottleneck features for speech synthesis,”
Z. Wu, C. Valentini-Botinhao, O. Watts, and S. King, · 2015
Later among the works it cites.
“Sentence-level control vectors for deep neural network speech synthesis,”
O. Watts, Z. Wu, and S. King, · 2015
Later among the works it cites.
“Multi-speaker modeling and speaker adaptation for DNN-based TTS synthesis,”
Y. Fan, Y. Qian, F. Soong, and L. He, · 2015
Later among the works it cites.
“The effect of neural networks in statistical parametric speech synthesis,”
K. Hashimoto, K. Oura, Y. Nankaku, and K. Tokuda, · 2015
Later among the works it cites.
“Modelling acoustic feature dependencies with artificial neural networks: Trajectory-RNADE,”
B. Uria, I. Murray, S. Renals, and C. Valentini-Botinhao, · 2015
Later among the works it cites.
“Acoustic modeling for speech synthesis: from HMM to RNN,” http://research.google.com/pubs/pub44630.html
H. Zen, · 2015
Later among the works it cites.
“Vocaine the vocoder and applications is speech synthesis,”
Y. Agiomyrgiannakis, · 2015
Later among the works it cites.
“Investigating gated recurrent neural networks for speech synthesis,”
Z. Wu and S. King, · 2016
Closest in time.
“From HMMs to DNNs: where do the improvements come from?,”
O. Watts, G. Henter, T. Merritt, Z. Wu, and S. King, · 2016
Closest in time.
“On the efficient representation and execution of deep acoustic models,”
R. Alvarez, R. Prabhavalkar, and A. Bakhtin, · 2016
Closest in time.
“Robust TTS duration modeling using DNNs,”
G. Henter, S. Ronanki, O. Watts, M. Wester, Z. Wu, and S. King, · 2016
Closest in time.
“Recent advances in Google real-time HMM-driven unit selection synthesizer,”
J. Gonzalvo, S. Tazari, C.-A. Chan, M. Becker, A. Gutkin, and H. Silen, · 2016
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…