2021

Guided-TTS: A Diffusion Model for Text-to-Speech via Classifier Guidance

Kim, Heeseung, Kim, Sungwon, Yoon, Sungroh

Understand

We propose Guided-TTS, a high-quality text-to-speech (TTS) model that does not require any transcript of target speaker using classifier guidance.

Guided-TTS combines an unconditional diffusion probabilistic model with a separately trained phoneme classifier for classifier guidance.
Our unconditional diffusion model learns to generate speech without any context from untranscribed speech data.
For TTS synthesis, we guide the generative process of the diffusion model with a phoneme classifier trained on a large-scale speech recognition dataset.

Built on

Reverse-time diffusion equation models
Anderson, B. D. O · 1982
Earlier work this paper cites.
Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks
Graves, A., Fernández, S., Gomez, F., and Schmidhuber, J · 2006
Earlier work this paper cites.
MNIST handwritten digit database
LeCun, Y. and Cortes, C · 2010
Earlier work this paper cites.
The blizzard challenge 2013
King, S. J. and Karaiskos, V · 2013
Earlier work this paper cites.
Generative adversarial nets
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y · 2014
Earlier work this paper cites.
Auto-encoding variational Bayes
Kingma, D. P. and Welling, M · 2014
Earlier work this paper cites.
Librispeech: An asr corpus based on public domain audio books
Panayotov, V., Chen, G., Povey, D., and Khudanpur, S · 2015
Earlier work this paper cites.
U-net: Convolutional networks for biomedical image segmentation
Ronneberger, O., Fischer, P., and Brox, T · 2015
Earlier work this paper cites.
Deep unsupervised learning using nonequilibrium thermodynamics
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S · 2015
Earlier work this paper cites.
Wavenet: A generative model for raw audio
Original
van den Oord, A., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A. W., and Kavukcuoglu, K · 2016
Earlier work this paper cites.
The lj speech dataset
Ito, K · 2017
Earlier work this paper cites.
Montreal forced aligner: Trainable text-speech alignment using kaldi
McAuliffe, M., Socolof, M., Mihuc, S., Wagner, M., and Sonderegger, M · 2017
Earlier work this paper cites.
Neural discrete representation learning
van den Oord, A., Vinyals, O., et al · 2017
Earlier work this paper cites.
Tacotron: Towards End-to-End Speech Synthesis
Wang, Y., Skerry-Ryan, R., Stanton, D., Wu, Y., Weiss, R. J., Jaitly, N., Yang, Z., Xiao, Y., Chen, Z., Bengio, S., Le, Q., Agiomyrgiannakis, Y., Clark, R., and Saurous, R. A · 2017
Earlier work this paper cites.
Neural voice cloning with a few samples
Arik, S., Chen, J., Peng, K., Ping, W., and Zhou, Y · 2018
Earlier work this paper cites.
Voxceleb2: Deep speaker recognition
Chung, J. S., Nagrani, A., and Zisserman, A · 2018
Earlier work this paper cites.
Transfer learning from speaker verification to multispeaker text-to-speech synthesis
Jia, Y., Zhang, Y., Weiss, R., Wang, Q., Shen, J., Ren, F., Chen, z., Nguyen, P., Pang, R., Lopez Moreno, I., and Wu, Y · 2018
Earlier work this paper cites.
Efficient neural audio synthesis
Kalchbrenner, N., Elsen, E., Simonyan, K., Noury, S., Casagrande, N., Lockhart, E., Stimberg, F., Oord, A., Dieleman, S., and Kavukcuoglu, K · 2018
Earlier work this paper cites.
Glow: Generative flow with invertible 1x1 convolutions
Kingma, D. P. and Dhariwal, P · 2018
Earlier work this paper cites.

Similar

Parallel wavenet: Fast high-fidelity speech synthesis
Oord, A., Li, Y., Babuschkin, I., Simonyan, K., Vinyals, O., Kavukcuoglu, K., Driessche, G., Lockhart, E., Cobo, L., Stimberg, F., et al · 2018
Cited alongside, same era.
Deep voice 3: Scaling text-to-speech with convolutional sequence learning
Ping, W., Peng, K., Gibiansky, A., Arik, S. Ö., Kannan, A., Narang, S., Raiman, J., and Miller, J · 2018
Cited alongside, same era.
Natural tts synthesis by conditioning wavenet on mel spectrogram predictions
Shen, J., Pang, R., Weiss, R. J., Schuster, M., Jaitly, N., Yang, Z., Chen, Z., Zhang, Y., Wang, Y., Skerrv-Ryan, R., et al · 2018
Cited alongside, same era.
Generalized end-to-end loss for speaker verification
Wan, L., Wang, Q., Papir, A., and Moreno, I. L · 2018
Cited alongside, same era.
High Fidelity Speech Synthesis with Adversarial Networks
Bińkowski, M., Donahue, J., Dieleman, S., Clark, A., Elsen, E., Casagrande, N., Cobo, L. C., and Simonyan, K · 2019
Cited alongside, same era.
Adversarial audio synthesis
Donahue, C., McAuley, J., and Puckette, M · 2019
Cited alongside, same era.
Flowavenet: A generative flow for raw audio
Kim, S., Lee, S.-G., Song, J., Kim, J., and Yoon, S · 2019
Cited alongside, same era.
Nemo: a toolkit for building ai applications using neural modules
Original
Kuchaiev, O., Li, J., Nguyen, H., Hrinchuk, O., Leary, R., Ginsburg, B., Kriman, S., Beliaev, S., Lavrukhin, V., Cook, J., et al · 2019
Cited alongside, same era.
Melgan: Generative adversarial networks for conditional waveform synthesis
Kumar, K., Kumar, R., de Boissiere, T., Gestin, L., Teoh, W. Z., Sotelo, J., de Brébisson, A., Bengio, Y., and Courville, A. C · 2019
Cited alongside, same era.
Neural speech synthesis with transformer network
Li, N., Liu, S., Liu, Y., Zhao, S., and Liu, M · 2019
Cited alongside, same era.
Waveglow: A flow-based generative network for speech synthesis
Prenger, R., Valle, R., and Catanzaro, B · 2019
Cited alongside, same era.
FastSpeech: Fast, Robust and Controllable Text to Speech
Ren, Y., Ruan, Y., Tan, X., Qin, T., Zhao, S., Zhao, Z., and Liu, T.-Y · 2019
Cited alongside, same era.

Then

Hi-fi multi-speaker english tts dataset
Original
Bakhturina, E., Lavrukhin, V., Ginsburg, B., and Zhang, Y · 2021
Closest in time.
Phonemizer
Bernard, M · 2021
Closest in time.
SC-GlowTTS: An Efficient Zero-Shot Multi-Speaker Text-To-Speech Model
Casanova, E., Shulby, C., Gölge, E., Müller, N. M., de Oliveira, F. S., Candido Jr., A., da Silva Soares, A., Aluisio, S. M., and Ponti, M. A · 2021
Closest in time.
WaveGrad 2: Iterative Refinement for Text-to-Speech Synthesis
Chen, N., Zhang, Y., Zen, H., Weiss, R. J., Norouzi, M., Dehak, N., and Chan, W · 2021
Closest in time.
Diffusion models beat gans on image synthesis
Original
Dhariwal, P. and Nichol, A · 2021
Closest in time.
End-to-end Adversarial Text-to-Speech
Donahue, J., Dieleman, S., Binkowski, M., Elsen, E., and Simonyan, K · 2021
Closest in time.
Diff-TTS: A Denoising Diffusion Model for Text-to-Speech
Jeong, M., Kim, H., Cheon, S. J., Choi, B. J., and Kim, N. S · 2021
Closest in time.
Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech
Original
Kim, J., Kong, J., and Son, J · 2021
Closest in time.
DiffWave: A Versatile Diffusion Model for Audio Synthesis
Kong, Z., Ping, W., Huang, J., Zhao, K., and Catanzaro, B · 2021
Closest in time.
On Generative Spoken Language Modeling from Raw Audio
Lakhotia, K., Kharitonov, E., Hsu, W.-N., Adi, Y., Polyak, A., Bolte, B., Nguyen, T.-A., Copet, J., Baevski, A., Mohamed, A., and Dupoux, E · 2021
Closest in time.
Vara-tts: Non-autoregressive text-to-speech synthesis based on very deep vae with residual attention
Original
Liu, P., Cao, Y., Liu, S., Hu, N., Li, G., Weng, C., and Su, D · 2021
Closest in time.
Diffusion probabilistic models for 3d point cloud generation
Luo, S. and Hu, W · 2021
Closest in time.
Improved denoising diffusion probabilistic models
Nichol, A. Q. and Dhariwal, P · 2021
Closest in time.
Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech
Popov, V., Vovk, I., Gogoryan, V., Sadekova, T., and Kudinov, M · 2021
Closest in time.
FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
Ren, Y., Hu, C., Tan, X., Qin, T., Zhao, S., Zhao, Z., and Liu, T.-Y · 2021
Closest in time.
Rad-tts: Parallel flow-based tts with robust alignment learning and diverse synthesis
Shih, K. J., Valle, R., Badlani, R., Lancucki, A., Ping, W., and Catanzaro, B · 2021
Closest in time.
Wave-tacotron: Spectrogram-free end-to-end text-to-speech synthesis
Weiss, R. J., Skerry-Ryan, R., Battenberg, E., Mariooryad, S., and Kingma, D. P · 2021
Closest in time.
Adaspeech 2: Adaptive text to speech with untranscribed data
Yan, Y., Tan, X., Li, B., Qin, T., Zhao, S., Shen, Y., and Liu, T.-Y · 2021
Closest in time.
Text-free prosody-aware generative spoken language modeling
Kharitonov, E., Lee, A., Polyak, A., Adi, Y., Copet, J., Lakhotia, K., Nguyen, T. A., Riviere, M., Mohamed, A., Dupoux, E., and Hsu, W.-N · 2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…