2019

MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis

Kumar, Kundan, Kumar, Rithesh, de Boissiere, Thibault et al.

Understand

Previous works (Donahue et al., 2018a; Engel et al., 2019a) have found that generating coherent raw audio waveforms with GANs is challenging.

In this paper, we show that it is possible to train GANs reliably to generate high quality coherent waveforms by introducing a set of architectural changes and simple training techniques.
Subjective evaluation metric (Mean Opinion Score, or MOS) shows the effectiveness of the proposed approach for high quality mel-spectrogram inversion.
To establish the generality of the proposed techniques, we show qualitative results of our model in speech synthesis, music domain translation and unconditional music synthesis.

Built on

Gansynth: Adversarial neural audio synthesis
Original
Engel, J., Agrawal, K. K., Chen, S., Gulrajani, I., Donahue, C., and Roberts, A · 1902
Earlier work this paper cites.
Gansynth: Adversarial neural audio synthesis
Original
Engel, J., Agrawal, K. K., Chen, S., Gulrajani, I., Donahue, C., and Roberts, A · 1902
Earlier work this paper cites.
Signal estimation from modified short-time fourier transform
Griffin, D. and Lim, J · 1984
Earlier work this paper cites.
Generative adversarial nets
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y · 2014
Earlier work this paper cites.
Autoencoding beyond pixels using a learned similarity metric
Original
Larsen, A. B. L., Sønderby, S. K., Larochelle, H., and Winther, O · 2015
Earlier work this paper cites.
Deep multi-scale video prediction beyond mean square error
Original
Mathieu, M., Couprie, C., and LeCun, Y · 2015
Earlier work this paper cites.
Generating images with perceptual similarity metrics based on deep networks
Dosovitskiy, A. and Brox, T · 2016
Earlier work this paper cites.
Image style transfer using convolutional neural networks
Gatys, L. A., Ecker, A. S., and Bethge, M · 2016
Earlier work this paper cites.
Perceptual losses for real-time style transfer and super-resolution
Johnson, J., Alahi, A., and Fei-Fei, L · 2016
Earlier work this paper cites.
Samplernn: An unconditional end-to-end neural audio generation model
Original
Mehri, S., Kumar, K., Gulrajani, I., Kumar, R., Jain, S., Sotelo, J., Courville, A., and Bengio, Y · 2016
Earlier work this paper cites.
World: A vocoder-based high-quality speech synthesis system for real-time applications
MORISE, M., YOKOMORI, F., and OZAWA, K · 2016
Earlier work this paper cites.
Deconvolution and checkerboard artifacts
Odena, A., Dumoulin, V., and Olah, C · 2016
Earlier work this paper cites.
Weight normalization: A simple reparameterization to accelerate training of deep neural networks
Salimans, T. and Kingma, D. P · 2016
Earlier work this paper cites.
Instance normalization: The missing ingredient for fast stylization
Original
Ulyanov, D., Vedaldi, A., and Lempitsky, V · 2016
Earlier work this paper cites.
Wavenet: A generative model for raw audio
Van Den Oord, A., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A. W., and Kavukcuoglu, K · 2016
Earlier work this paper cites.
Deep voice: Real-time neural text-to-speech
Arik, S. Ö., Chrzanowski, M., Coates, A., Diamos, G., Gibiansky, A., Kang, Y., Li, X., Miller, J., Ng, A., Raiman, J., et al · 2017
Earlier work this paper cites.

Similar

Neural audio synthesis of musical notes with wavenet autoencoders
Engel, J., Resnick, C., Roberts, A., Dieleman, S., Norouzi, M., Eck, D., and Simonyan, K · 2017
Cited alongside, same era.
Deep voice 2: Multi-speaker neural text-to-speech
Gibiansky, A., Arik, S., Diamos, G., Miller, J., Peng, K., Ping, W., Raiman, J., and Zhou, Y · 2017
Cited alongside, same era.
Improved training of wasserstein gans
Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., and Courville, A. C · 2017
Cited alongside, same era.
Image-to-image translation with conditional adversarial networks
Isola, P., Zhu, J.-Y., Zhou, T., and Efros, A. A · 2017
Cited alongside, same era.
The lj speech dataset
Ito, K · 2017
Cited alongside, same era.
Progressive growing of gans for improved quality, stability, and variation
Original
Karras, T., Aila, T., Laine, S., and Lehtinen, J · 2017
Cited alongside, same era.
Geometric gan
Original
Lim, J. H. and Ye, J. C · 2017
Cited alongside, same era.
Least squares generative adversarial networks
Mao, X., Li, Q., Xie, H., Lau, R. Y., Wang, Z., and Paul Smolley, S · 2017
Cited alongside, same era.
Parallel wavenet: Fast high-fidelity speech synthesis
Original
Oord, A. v. d., Li, Y., Babuschkin, I., Simonyan, K., Vinyals, O., Kavukcuoglu, K., Driessche, G. v. d., Lockhart, E., Cobo, L. C., Stimberg, F., et al · 2017
Cited alongside, same era.
Deep voice 3: Scaling text-to-speech with convolutional sequence learning
Original
Ping, W., Peng, K., Gibiansky, A., Arik, S. O., Kannan, A., Narang, S., Raiman, J., and Miller, J · 2017
Cited alongside, same era.
Char2wav: End-to-end speech synthesis
Sotelo, J., Mehri, S., Kumar, K., Santos, J. F., Kastner, K., Courville, A., and Bengio, Y · 2017
Cited alongside, same era.
Neural discrete representation learning
van den Oord, A., Vinyals, O., et al · 2017
Cited alongside, same era.

Then

The challenge of realistic music generation: modelling raw audio at scale
Dieleman, S., van den Oord, A., and Simonyan, K · 2018
Later among the works it cites.
Dissecting the nvidia volta gpu architecture via microbenchmarking
Original
Jia, Z., Maggioni, M., Staiger, B., and Scarpazza, D. P · 2018
Later among the works it cites.
Efficient neural audio synthesis
Original
Kalchbrenner, N., Elsen, E., Simonyan, K., Noury, S., Casagrande, N., Lockhart, E., Stimberg, F., Oord, A. v. d., Dieleman, S., and Kavukcuoglu, K · 2018
Later among the works it cites.
A style-based generator architecture for generative adversarial networks
Original
Karras, T., Laine, S., and Aila, T · 2018
Later among the works it cites.
Glow: Generative flow with invertible 1x1 convolutions
Kingma, D. P. and Dhariwal, P · 2018
Later among the works it cites.
Spectral normalization for generative adversarial networks
Original
Miyato, T., Kataoka, T., Koyama, M., and Yoshida, Y · 2018
Later among the works it cites.
Clarinet: Parallel wave generation in end-to-end text-to-speech
Original
Ping, W., Peng, K., and Chen, J · 2018
Later among the works it cites.
Natural tts synthesis by conditioning wavenet on mel spectrogram predictions
Shen, J., Pang, R., Weiss, R. J., Schuster, M., Jaitly, N., Yang, Z., Chen, Z., Zhang, Y., Wang, Y., Skerrv-Ryan, R., et al · 2018
Later among the works it cites.
Invariances and data augmentation for supervised music transcription
Thickstun, J., Harchaoui, Z., Foster, D. P., and Kakade, S. M · 2018
Later among the works it cites.
Self-attention generative adversarial networks
Original
Zhang, H., Goodfellow, I., Metaxas, D., and Odena, A · 2018
Later among the works it cites.
Autoencoder-based music translation
Mor, N., Wolf, L., Polyak, A., and Taigman, Y · 2019
Closest in time.
Expediting tts synthesis with adversarial vocoding
Original
Neekhara, P., Donahue, C., Puckette, M., Dubnov, S., and McAuley, J · 2019
Closest in time.
Semantic image synthesis with spatially-adaptive normalization
Original
Park, T., Liu, M.-Y., Wang, T.-C., and Zhu, J.-Y · 2019
Closest in time.
Waveglow: A flow-based generative network for speech synthesis
Prenger, R., Valle, R., and Catanzaro, B · 2019
Closest in time.
r9y9/wavenet_vocoder, Oct 2019
Yamamoto, R · 2019
Closest in time.
Probability density distillation with generative adversarial networks for high-quality parallel waveform generation
Original
Yamamoto, R., Song, E., and Kim, J.-M · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…