2018

Deep Encoder-Decoder Models for Unsupervised Learning of Controllable Speech Synthesis

Henter, Gustav Eje, Lorenzo-Trueba, Jaime, Wang, Xin et al.

Understand

Generating versatile and appropriate synthetic speech requires control over the output expression separate from the spoken text.

Important non-textual speech variation is seldom annotated, in which case output control must be learned in an unsupervised fashion.
In this paper, we perform an in-depth study of methods for unsupervised learning of control in statistical speech synthesis.
For example, we show that popular unsupervised training heuristics can be interpreted as variational inference in certain autoencoder models.

Built on

Y. Fan, Y. Qian, F.-L. Xie, and F. K. Soong, “TTS synthesis with bidirectional LSTM based recurrent neural networks,” in Proc. Interspeech , 2014, pp. 1964–1968
1968
Earlier work this paper cites.
A. P. Dempster, N. M. Laird, and D. B. Rubin, “Maximum likelihood from incomplete data via the EM algorithm,” J. Roy. Stat. Soc. B , vol. 39, no. 1, pp. 1–38, 1977
1977
Earlier work this paper cites.
S. Holm, “A simple sequentially rejective multiple test procedure,” Scand. J. Stat. , vol. 6, no. 2, pp. 65–70, 1979
1979
Earlier work this paper cites.
D. H. Klatt, “Review of text-to-speech conversion for English,” The Journal of the Acoustical Society of America , vol. 82, no. 3, pp. 737–793, 1987
1987
Earlier work this paper cites.
L. R. Rabiner, “A tutorial on hidden Markov models and selected applications in speech recognition,” Proc. IEEE , vol. 77, no. 2, pp. 257–286, 1989
1989
Earlier work this paper cites.
P. Dayan, G. E. Hinton, R. M. Neal, and R. S. Zemel, “The Helmholtz machine,” Neural Comput. , vol. 7, no. 5, pp. 889–904, 1995
1995
Earlier work this paper cites.
J. R. Quinlan, “Improved use of continuous attributes in C4.5,” J. Artif. Intel. Res. , vol. 4, pp. 77–90, 1996
1996
Earlier work this paper cites.
M. J. F. Gales, “Cluster adaptive training of hidden Markov models,” IEEE T. Speech Audi. P. , vol. 8, no. 4, pp. 417–428, 2000
2000
Earlier work this paper cites.
K. Tokuda, T. Yoshimura, T. Masuko, T. Kobayashi, and T. Kitamura, “Speech parameter generation algorithms for HMM-based speech synthesis,” in Proc. ICASSP , 2000, pp. 1315–1318
2000
Earlier work this paper cites.
K. Fujinaga, M. Nakai, H. Shimodaira, and S. Sagayama, “Multiple-regression hidden Markov model,” in Proc. ICASSP , 2001, pp. 513–516
2001
Earlier work this paper cites.
T. Masuko, T. Kobayashi, and K. Miyanaga, “A style control technique for HMM-based speech synthesis,” in Proc. Interspeech , 2004, pp. 1437–1439
2004
Earlier work this paper cites.
J. Yamagishi, K. Onishi, T. Masuko, and T. Kobayashi, “Acoustic modeling of speaking styles and emotional expressions in HMM-based speech synthesis,” IEICE T. Inf. Syst. , vol. 88, no. 3, pp. 502–509, 2005
2005
Earlier work this paper cites.
T. Yoshimura, K. Tokuda, T. Masuko, T. Kobayashi, and T. Kitamura, “Incorporating a mixed excitation model and postfilter into HMM-based text-to-speech synthesis,” Syst. Comput. Jpn. , vol. 36, no. 12, pp. 43–50, 2005
2005
Earlier work this paper cites.
C. M. Bishop, Pattern Recognition and Machine Learning , 1st ed. New York, NY: Springer, 2006
2006
Earlier work this paper cites.
H. Kawahara, “STRAIGHT, exploitation of the other aspect of VOCODER: Perceptually isomorphic decomposition of speech sounds,” Acoust. Sci. Technol. , vol. 27, no. 6, pp. 349–353, 2006
2006
Earlier work this paper cites.
T. Nose, Y. Kato, and T. Kobayashi, “Style estimation of speech based on multiple regression hidden semi-Markov model,” in Proc. Interspeech , 2007, pp. 2285–2288
2007
Earlier work this paper cites.
H. Zen, T. Nose, J. Yamagishi, S. Sako, T. Masuko, A. W. Black, and K. Tokuda, “The HMM-based speech synthesis system (HTS) version 2.0,” in Proc. SSW , 2007, pp. 294–299
2007
Earlier work this paper cites.
A. Camacho and J. G. Harris, “A sawtooth waveform inspired pitch estimator for speech and music,” The Journal of the Acoustical Society of America , vol. 124, no. 3, pp. 1638–1652, 2008
2008
Earlier work this paper cites.
L. van der Maaten and G. Hinton, “Visualizing data using t-SNE,” J. Mach. Learn. Res. , vol. 9, no. Nov, pp. 2579–2605, 2008
2008
Earlier work this paper cites.
C. D. Manning, P. Raghavan, and H. Schütze, Introduction to Information Retrieval . Cambridge University Press, 2008
2008
Earlier work this paper cites.
H. Zen, K. Tokuda, and A. W. Black, “Statistical parametric speech synthesis,” Speech Commun. , vol. 51, no. 11, pp. 1039–1064, 2009
2009
Earlier work this paper cites.
D. Povey, L. Burget, M. Agarwal, P. Akyazi, K. Feng, A. Ghoshal, O. Glembek, N. K. Goel, M. Karafiát, A. Rastrow, R. C. Rose, P. Schwarz, and S. Thomas, “Subspace Gaussian mixture models for speech recognition,” in Proc. ICASSP , 2010, pp. 4330–4333
2010
Earlier work this paper cites.
R. Barra-Chicote, J. Yamagishi, S. King, J. M. Montero, and J. Macias-Guarasa, “Analysis of statistical parametric and unit selection speech synthesis systems applied to emotional speech,” Speech Commun. , vol. 52, no. 5, pp. 394–404, 2010
2010
Earlier work this paper cites.
D. Erro, E. Navas, I. Herndez, and I. Saratxaga, “Emotion conversion based on prosodic unit selection,” IEEE T. Audio Speech , vol. 18, no. 5, pp. 974–983, 2010
2010
Earlier work this paper cites.
K. Oura, S. Sako, and K. Tokuda, “Japanese text-to-speech synthesis system: Open JTalk,” in Proc. ASJ Spring , 2010, pp. 343–344
2010
Earlier work this paper cites.
X. Glorot and Y. Bengio, “Understanding the difficulty of training deep feedforward neural networks,” in Proc. AISTATS , 2010, pp. 249–256
2010
Earlier work this paper cites.
S. King, “An introduction to statistical parametric speech synthesis,” Sadhana , vol. 36, no. 5, pp. 837–852, 2011
2011
Earlier work this paper cites.
L. Chen, M. J. F. Gales, V. Wan, J. Latorre, and M. Akamine, “Exploring rich expressive information from audiobook data using cluster adaptive training,” in Proc. Interspeech , 2012, pp. 959–962
2012
Earlier work this paper cites.
O. Abdel-Hamid and H. Jiang, “Fast speaker adaptation of hybrid NN/HMM model for speech recognition based on discriminative learning of speaker code,” in Proc. ICASSP , 2013, pp. 7942–7946
2013
Earlier work this paper cites.
Z.-H. Ling, K. Richmond, and J. Yamagishi, “Articulatory control of HMM-based parametric speech synthesis using feature-space-switched multiple regression,” IEEE T. Audio Speech , vol. 21, no. 1, pp. 207–219, 2013
2013
Earlier work this paper cites.
T. Nose and T. Kobayashi, “An intuitive style control technique in HMM-based expressive speech synthesis using subjective style intensity and multiple-regression global variance model,” Speech Commun. , vol. 55, no. 2, pp. 347–357, 2013
2013
Earlier work this paper cites.
Y. Bengio, N. Léonard, and A. Courville, “Estimating or propagating gradients through stochastic neurons for conditional computation,” arXiv preprint arXiv:1308.3432 , 2013
Original
2013
Earlier work this paper cites.
G. E. Henter, T. Merritt, M. Shannon, C. Mayo, and S. King, “Measuring the perceptual effects of modelling assumptions in speech synthesis using stimuli constructed from repeated natural speech,” in Proc. Interspeech , 2014, pp. 1504–1508
2014
Earlier work this paper cites.
S. Xue, O. Abdel-Hamid, H. Jiang, L.-R. Dai, and Q. Liu, “Fast adaptation of deep neural network based on discriminant codes for speech recognition,” IEEE/ACM T. Audio Speech , vol. 22, no. 12, pp. 1713–1725, 2014
2014
Earlier work this paper cites.

Similar

Q. V. Le and T. Mikolov, “Distributed representations of sentences and documents,” in Proc. ICML , 2014, pp. 1188–1196
2014
Cited alongside, same era.
D. P. Kingma and M. Welling, “Auto-encoding variational Bayes,” in Proc. ICLR , 2014
2014
Cited alongside, same era.
D. J. Rezende, S. Mohamed, and D. Wierstra, “Stochastic backpropagation and approximate inference in deep generative models,” in Proc. ICML , vol. 32, no. 2, 2014, pp. 1278–1286
2014
Cited alongside, same era.
I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in Proc. NIPS , 2014, pp. 2672–2680
2014
Cited alongside, same era.
O. Fabius and J. R. van Amersfoort, “Variational recurrent auto-encoders,” Proc. ICLR Workshop Track , 2014
2014
Cited alongside, same era.
B. Uria, I. Murray, S. Renals, C. Valentini-Botinhao, and J. Bridle, “Modelling acoustic feature dependencies with artificial neural networks: Trajectory-RNADE,” in Proc. ICASSP , 2015, pp. 4465–4469
2015
Cited alongside, same era.
Y. Fan, Y. Qian, F. K. Soong, and L. He, “Multi-speaker modeling and speaker adaptation for DNN-based TTS synthesis,” in Proc. ICASSP , 2015, pp. 4475–4479
2015
Cited alongside, same era.
O. Watts, Z. Wu, and S. King, “Sentence-level control vectors for deep neural network speech synthesis,” in Proc. Interspeech , 2015, pp. 2217–2221
2015
Cited alongside, same era.
J. Chung, K. Kastner, L. Dinh, K. Goel, A. Courville, and Y. Bengio, “A recurrent latent variable model for sequential data,” in Proc. NIPS , 2015, pp. 2980–2988
2015
Cited alongside, same era.
J. Lorenzo-Trueba, R. Barra-Chicote, R. San-Segundo, J. Ferreiros, J. Yamagishi, and J. M. Montero, “Emotion transplantation through adaptation in HMM-based speech synthesis,” Comput. Speech Lang. , 2015
2015
Cited alongside, same era.
M. Morise, “Cheaptrick, a spectral envelope estimator for high-quality speech synthesis,” Speech Commun. , vol. 67, pp. 1–7, 2015
2015
Cited alongside, same era.
D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in Proc. ICLR , 2015
2015
Cited alongside, same era.

Then

A. van den Oord, O. Vinyals, and K. Kavukcuoglu, “Neural discrete representation learning,” in Proc. NIPS , 2017, pp. 6309–6318
2017
Later among the works it cites.
G. E. Henter, J. Lorenzo-Trueba, X. Wang, and J. Yamagishi, “Principles for learning controllable TTS from annotated and latent variation,” in Proc. Interspeech , 2017, pp. 3956–3960
2017
Later among the works it cites.
I. Jauk, “Unsupervised learning for expressive speech synthesis,” Ph.D. dissertation, Polytechnic University of Catalonia, Barcelona, Spain, Jun 2017
2017
Later among the works it cites.
K. Sawada, K. Hashimoto, K. Oura, and K. Tokuda, “The NITech text-to-speech system for the Blizzard Challenge 2017,” in Proc. Blizzard Challenge Workshop , 2017
2017
Later among the works it cites.
S. King, L. Wihlborg, and W. Guo, “The Blizzard Challenge 2017,” in Proc. Blizzard Challenge Workshop , 2017
2017
Later among the works it cites.
Y. Wang, R. Skerry-Ryan, D. Stanton, Y. Wu, R. J. Weiss, N. Jaitly, Z. Yang, Y. Xiao, Z. Chen, S. Bengio, Q. Le, Y. Agiomyrgiannakis, R. Clark, and R. A. Saurous, “Tacotron: A fully end-to-end text-to-speech synthesis model,” in Proc. Interspeech , 2017, pp. 4006–4010
2017
Later among the works it cites.
Y. Wang, R. Skerry-Ryan, Y. Xiao, D. Stanton, J. Shor, E. Battenberg, R. Clark, and R. A. Saurous, “Uncovering latent style factors for expressive speech synthesis,” in NIPS ML4Audio Workshop , 2017
2017
Later among the works it cites.
F. Huszár. (2017) Is maximum likelihood useful for representation learning? [Online]. Available: http://www.inference.vc/maximum-likelihood-for-representation-learning-2/
2017
Later among the works it cites.
——, “Voice conversion from unaligned corpora using variational autoencoding Wasserstein generative adversarial networks,” in Proc. Interspeech , 2017, pp. 3364–3368
2017
Later among the works it cites.
W.-N. Hsu, Y. Zhang, and J. Glass, “Learning latent representations for speech generation and transformation,” in Proc. Interspeech , 2017, p. 1273–1277
2017
Later among the works it cites.
——, “Unsupervised learning of disentangled and interpretable representations from sequential data,” in Proc. NIPS , 2017, pp. 1878–1889
2017
Later among the works it cites.
S. Mehri, K. Kumar, I. Gulrajani, R. Kumar, S. Jain, J. Sotelo, A. Courville, and Y. Bengio, “SampleRNN: An unconditional end-to-end neural audio generation model,” in Proc. ICLR , 2017
2017
Later among the works it cites.
J. Sotelo, S. Mehri, K. Kumar, J. a. F. Santos, K. Kastner, A. Courville, and Y. Bengio, “Char2Wav: End-to-end speech synthesis,” in Proc. ICLR Workshop Track , 2017
2017
Later among the works it cites.
M. D. Hoffman, C. Riquelme, and M. J. Johnson, “The β \beta -vae’s implicit prior,” in Proc. NIPS 2017 Workshop Bayesian Deep Learn. , vol. 2, 2017
2017
Later among the works it cites.
J. M. Tomczak and M. Welling, “VAE with a VampPrior,” arXiv preprint arXiv:1705.07120 , 2017
Original
2017
Later among the works it cites.
X. Wang, S. Takaki, and J. Yamagishi, “An autoregressive recurrent mixture density network for parametric speech synthesis,” in Proc. ICASSP , 2017, pp. 4895–4899
2017
Later among the works it cites.
Y. Taigman, L. Wolf, A. Polyak, and E. Nachmani, “VoiceLoop: Voice fitting and synthesis via a phonological loop,” in Proc. ICLR , 2018
2018
Closest in time.
R. Skerry-Ryan, E. Battenberg, Y. Xiao, Y. Wang, D. Stanton, J. Shor, R. J. Weiss, R. Clark, and R. A. Saurous, “Towards end-to-end prosody transfer for expressive speech synthesis with Tacotron,” arXiv preprint arXiv:1803.09047 , 2018
Original
2018
Closest in time.
Y. Wang, D. Stanton, Y. Zhang, R. Skerry-Ryan, E. Battenberg, J. Shor, Y. Xiao, F. Ren, Y. Jia, and R. A. Saurous, “Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis,” arXiv preprint arXiv:1803.09017 , 2018
Original
2018
Closest in time.
Y. Jia, Y. Zhang, R. J. Weiss, Q. Wang, J. Shen, F. Ren, Z. Chen, P. Nguyen, R. Pang, I. L. Moreno, and W. Yonghui, “Transfer learning from speaker verification to multispeaker text-to-speech synthesis,” arXiv preprint arXiv:1806.04558 , 2018
Original
2018
Closest in time.
A. Graves, J. Menick, and A. van den Oord, “Associative compression networks for representation learning,” arXiv preprint arXiv:1804.02476 , 2018
Original
2018
Closest in time.
H. Kameoka, T. Kaneko, K. Tanaka, and N. Hojo, “ACVAE-VC: Non-parallel many-to-many voice conversion with auxiliary classifier variational autoencoder,” arXiv preprint arXiv:1808.05092 , 2018
Original
2018
Closest in time.
K. Akuzawa, Y. Iwasawa, and Y. Matsuo, “Expressive speech synthesis via modeling expressions with variational autoencoder,” in Proc. Interspeech , 2018, to appear
2018
Closest in time.
J. Marino, M. Cvitkovic, and Y. Yue, “A general framework for amortizing variational filtering,” in ICML 2018 Workshop Theor. Found. Appl. Deep Gener. Model. , 2018
2018
Closest in time.
N. Kalchbrenner, E. Elsen, K. Simonyan, S. Noury, N. Casagrande, E. Lockhart, F. Stimberg, A. van den Oord, S. Dieleman, and K. Kavukcuoglu, “Efficient neural audio synthesis,” in Proc. ICML , 2018, pp. 2410–2419
2018
Closest in time.
X. Wang, “Fundamental frequency modeling for neural-network-based statistical parametric speech synthesis,” Ph.D. dissertation, SOKENDAI (The Graduate University for Advanced Studies), Tokyo, Japan, Sep 2018
2018
Closest in time.
X. Wang, S. Takaki, and J. Yamagishi, “Autoregressive neural F0 model for statistical parametric speech synthesis,” IEEE/ACM T. Audio Speech , vol. 26, no. 8, pp. 1406–1419, 2018
2018
Closest in time.
J. Lorenzo-Trueba, G. E. Henter, S. Takaki, J. Yamagishi, Y. Morino, and Y. Ochiai, “Investigating different representations for modeling and controlling multiple emotions in dnn-based speech,” Speech Commun. , 2018
2018
Closest in time.
W. Ping, K. Peng, A. Gibiansky, S. Ö. Arık, A. Kannan, S. Narang, J. Raiman, and J. Miller, “Deep Voice 3: Scaling text-to-speech with convolutional sequence learning,” in Proc. ICLR , 2018
2018
Closest in time.
J. Shen, R. Pang, R. J. Weiss, M. Schuster, N. Jaitly, Z. Yang, Z. Chen, Y. Zhang, Y. Wang, R. Skerry-Ryan, R. A. Saurous, Y. Agiomyrgiannakis, and Y. Wu, “Natural TTS synthesis by conditioning WaveNet on mel spectrogram predictions,” in Proc. ICASSP , 2018, pp. 4799–4783
2018
Closest in time.
C. Cremer, X. Li, and D. Duvenaud, “Inference suboptimality in variational autoencoders,” in Proc. ICLR Workshop Track , 2018
2018
Closest in time.
R. Shu, H. H. Bui, S. Zhao, M. J. Kochenderfer, and S. Ermon, “Amortized inference regularization,” arXiv preprint arXiv:1805.08913 , 2018
Original
2018
Closest in time.
E. Nachmani, A. Polyak, Y. Taigman, and L. Wolf, “Fitting new speakers based on a short untranscribed sample,” in Proc. ICML , 2018, pp. 3683–3691
2018
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…