2020

Multi-SpectroGAN: High-Diversity and High-Fidelity Spectrogram Generation with Adversarial Style Combination for Speech Synthesis

Lee, Sang-Hoon, Yoon, Hyun-Wook, Noh, Hyeong-Rae et al.

Understand

While generative adversarial networks (GANs) based neural text-to-speech (TTS) systems have shown significant improvement in neural speech synthesis, there is no TTS system to learn to synthesize speech from text sequences with only adversarial feedback.

Because adversarial feedback alone is not sufficient to train the generator, current models still require the reconstruction loss compared with the ground-truth and the generated mel-spectrogram directly.
In this paper, we present Multi-SpectroGAN (MSG), which can train the multi-speaker model with only the adversarial feedback by conditioning a self-supervised hidden representation of the generator to a conditional discriminator.
This leads to better guidance for generator training.

Built on

High fidelity speech synthesis with adversarial networks
Original
Bińkowski, M.; Donahue, J.; Dieleman, S.; Clark, A.; Elsen, E.; Casagrande, N.; Cobo, L. C.; and Simonyan, K. 2019 · 1909
Earlier work this paper cites.
Mel-cepstral distance measure for objective speech quality assessment
Kubichek, R. 1993 · 1993
Earlier work this paper cites.
Integrated segmentation and recognition of handwritten numerals with cascade neural network
Lee, S.-W.; and Kim, S.-Y. 1999 · 1999
Earlier work this paper cites.
Biologically motivated computer vision
Bulthoff, H. H.; Lee, S.-W.; Poggio, T.; and Wallraven, C. 2003 · 2003
Earlier work this paper cites.
MultiSpeech: Multi-Speaker Text to Speech with Transformer
Original
Chen, M.; Tan, X.; Ren, Y.; Xu, J.; Sun, H.; Zhao, S.; and Qin, T. 2020 · 2006
Earlier work this paper cites.
End-to-End Adversarial Text-to-Speech
Original
Donahue, J.; Dieleman, S.; Bińkowski, M.; Elsen, E.; and Simonyan, K. 2020 · 2006
Earlier work this paper cites.
FastPitch: Parallel Text-to-speech with Pitch Prediction
Original
Łańcucki, A. 2020 · 2006
Earlier work this paper cites.
FastSpeech 2: Fast and High-Quality End-to-End Text-to-Speech
Original
Ren, Y.; Hu, C.; Qin, T.; Zhao, S.; Zhao, Z.; and Liu, T.-Y. 2020 · 2006
Earlier work this paper cites.
Reconstruction of 3D human body pose from stereo image sequences based on top-down learning
Yang, H.-D.; and Lee, S.-W. 2007 · 2007
Earlier work this paper cites.
Audio Dequantization for High Fidelity Audio Generation in Flow-based Neural Vocoder
Original
Yoon, H.-W.; Lee, S.-H.; Noh, H.-R.; and Lee, S.-W. 2020 · 2008
Earlier work this paper cites.
Learning phrase representations using RNN encoder-decoder for statistical machine translation
Original
Cho, K.; Van Merriënboer, B.; Gulcehre, C.; Bahdanau, D.; Bougares, F.; Schwenk, H.; and Bengio, Y. 2014 · 2014
Earlier work this paper cites.
Generative adversarial nets
Goodfellow, I.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; and Bengio, Y. 2014 · 2014
Earlier work this paper cites.

Similar

Adam: A method for stochastic optimization
Kingma, D. P.; and Ba, J. 2015 · 2015
Cited alongside, same era.
Layer normalization
Original
Ba, J. L.; Kiros, J. R.; and Hinton, G. E. 2016 · 2016
Cited alongside, same era.
Wavenet: A generative model for raw audio
Original
Oord, A. v. d.; Dieleman, S.; Zen, H.; Simonyan, K.; Vinyals, O.; Graves, A.; Kalchbrenner, N.; Senior, A.; and Kavukcuoglu, K. 2016 · 2016
Cited alongside, same era.
Least squares generative adversarial networks
Mao, X.; Li, Q.; Xie, H.; Lau, R. Y.; Wang, Z.; and Paul Smolley, S. 2017 · 2017
Cited alongside, same era.
Attention is All you Need
Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017 · 2017
Cited alongside, same era.
Tacotron: Towards end-to-end speech synthesis
Original
Wang, Y.; Skerry-Ryan, R.; Stanton, D.; Wu, Y.; Weiss, R. J.; Jaitly, N.; Yang, Z.; Xiao, Y.; Chen, Z.; Bengio, S.; et al. 2017 · 2017
Cited alongside, same era.
mixup: Beyond empirical risk minimization
Original
Zhang, H.; Cisse, M.; Dauphin, Y. N.; and Lopez-Paz, D. 2017 · 2017
Cited alongside, same era.
FloWaveNet: A generative flow for raw audio
Original
Kim, S.; Lee, S.-g.; Song, J.; Kim, J.; and Yoon, S. 2018 · 2018
Cited alongside, same era.
Parallel wavenet: Fast high-fidelity speech synthesis
Oord, A.; Li, Y.; Babuschkin, I.; Simonyan, K.; Vinyals, O.; Kavukcuoglu, K.; Driessche, G.; Lockhart, E.; Cobo, L.; Stimberg, F.; et al. 2018 · 2018
Cited alongside, same era.
Natural tts synthesis by conditioning wavenet on mel spectrogram predictions
Shen, J.; Pang, R.; Weiss, R. J.; Schuster, M.; Jaitly, N.; Yang, Z.; Chen, Z.; Zhang, Y.; Wang, Y.; Skerrv-Ryan, R.; et al. 2018 · 2018
Cited alongside, same era.
Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron
Skerry-Ryan, R.; Battenberg, E.; Xiao, Y.; Wang, Y.; Stanton, D.; Shor, J.; Weiss, R.; Clark, R.; and Saurous, R. A. 2018 · 2018
Cited alongside, same era.

Then

Generalized end-to-end loss for speaker verification
Wan, L.; Wang, Q.; Papir, A.; and Moreno, I. L. 2018 · 2018
Later among the works it cites.
Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis
Original
Wang, Y.; Stanton, D.; Zhang, Y.; Skerry-Ryan, R.; Battenberg, E.; Shor, J.; Xiao, Y.; Ren, F.; Jia, Y.; and Saurous, R. A. 2018 · 2018
Later among the works it cites.
On adversarial mixup resynthesis
Beckham, C.; Honari, S.; Verma, V.; Lamb, A. M.; Ghadiri, F.; Hjelm, R. D.; Bengio, Y.; and Pal, C. 2019 · 2019
Later among the works it cites.
Melgan: Generative adversarial networks for conditional waveform synthesis
Kumar, K.; Kumar, R.; de Boissiere, T.; Gestin, L.; Teoh, W. Z.; Sotelo, J.; de Brébisson, A.; Bengio, Y.; and Courville, A. C. 2019 · 2019
Later among the works it cites.
Neural speech synthesis with transformer network
Li, N.; Liu, S.; Liu, Y.; Zhao, S.; and Liu, M. 2019 · 2019
Later among the works it cites.
Waveglow: A flow-based generative network for speech synthesis
Prenger, R.; Valle, R.; and Catanzaro, B. 2019 · 2019
Later among the works it cites.
Fastspeech: Fast, robust and controllable text to speech
Ren, Y.; Ruan, Y.; Tan, X.; Qin, T.; Zhao, S.; Zhao, Z.; and Liu, T.-Y. 2019 · 2019
Later among the works it cites.
Manifold mixup: Better representations by interpolating hidden states
Verma, V.; Lamb, A.; Beckham, C.; Najafi, A.; Mitliagkas, I.; Lopez-Paz, D.; and Bengio, Y. 2019 · 2019
Later among the works it cites.
Learning latent representations for style control and transfer in end-to-end speech synthesis
Original
Zhang, Y.-J.; Pan, S.; He, L.; and Ling, Z.-H. 2019 · 2019
Later among the works it cites.
RobuTrans: A Robust Transformer-Based Text-to-Speech Model
Li, N.; Liu, Y.; Wu, Y.; Liu, S.; Zhao, S.; and Liu, M. 2020 · 2020
Closest in time.
Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram
Yamamoto, R.; Song, E.; and Kim, J.-M. 2020 · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…