2020

WaveGrad: Estimating Gradients for Waveform Generation

Chen, Nanxin, Zhang, Yu, Zen, Heiga et al.

Understand

This paper introduces WaveGrad, a conditional model for waveform generation which estimates gradients of the data density.

The model is built on prior work on score matching and diffusion probabilistic models.
It starts from a Gaussian white noise signal and iteratively refines the signal via a gradient-based sampler conditioned on the mel-spectrogram.
WaveGrad offers a natural way to trade inference speed for sample quality by adjusting the number of refinement steps, and bridges the gap between non-autoregressive and autoregressive models in terms of audio quality.

Built on

KERMIT: Generative Insertion-Based Modeling for Sequences
Original
William Chan, Nikita Kitaev, Kelvin Guu, Mitchell Stern, and Jakob Uszkoreit · 1906
Earlier work this paper cites.
An Empirical Study of Generation Order for Machine Translation
Original
William Chan, Mitchell Stern, Jamie Kiros, and Jakob Uszkoreit · 1910
Earlier work this paper cites.
Mel-Cepstral Distance Measure for Objective Speech Quality Assessment
Robert Kubichek · 1993
Earlier work this paper cites.
Estimation of Non-Normalized Statistical Models by Score Matching
Aapo Hyvärinen · 2005
Earlier work this paper cites.
Multi-band MelGAN: Faster Waveform Generation for High-Quality Text-to-Speech
Original
Geng Yang, Shan Yang, Kai Liu, Peng Fang, Wei Chen, and Lei Xie · 2005
Earlier work this paper cites.
VocGAN: A High-Fidelity Real-time Vocoder with a Hierarchically-nested Adversarial Network
Original
Jinhyeok Yang, Junmo Lee, Youngik Kim, Hoonyoung Cho, and Injung Kim · 2007
Earlier work this paper cites.
Reducing F0 Frame Error of F0 Tracking Algorithms under Noisy Conditions with an Unvoiced/Voiced Classification Frontend
Wei Chu and Abeer Alwan · 2009
Earlier work this paper cites.
A Connection Between Score Matching and Denoising Autoencoders
Pascal Vincent · 2011
Earlier work this paper cites.
Exact Solutions to the Nonlinear Dynamics of Learning in Deep Linear Neural Networks
Andrew M. Saxe, James L. McClelland, and Surya Ganguli · 2014
Earlier work this paper cites.
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Sergey Ioffe and Christian Szegedy · 2015
Earlier work this paper cites.
Deep Unsupervised Learning using Nonequilibrium Thermodynamics
Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli · 2015
Earlier work this paper cites.
Deep Residual Learning for Image Recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2016
Earlier work this paper cites.
WaveNet: A Generative Model for Raw Audio
Original
Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu · 2016
Earlier work this paper cites.
The LJ Speech Dataset, 2017
Keith Ito and Linda Johnson · 2017
Earlier work this paper cites.
SampleRNN: An Unconditional End-to-End Neural Audio Generation Model
Soroush Mehri, Kundan Kumar, Ishaan Gulrajani, Rithesh Kumar, Shubham Jain, Jose Sotelo, Aaron Courville, and Yoshua Bengio · 2017
Earlier work this paper cites.
PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P. Kingma · 2017
Earlier work this paper cites.
Char2Wav: End-to-End Speech Synthesis
Jose Sotelo, Soroush Mehri, Kundan Kumar, Joao Felipe Santos, Kyle Kastner, Aaron C. Courville, and Yoshua Bengio · 2017
Earlier work this paper cites.
Attention Is All You Need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin · 2017
Earlier work this paper cites.
Tacotron: Towards End-to-End Speech Synthesis
Yuxuan Wang, RJ Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J. Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, Quoc Le, Yannis Agiomyrgiannakis, Rob Clark, and Rif A. Saurous · 2017
Earlier work this paper cites.
Adversarial Audio Synthesis
Original
Chris Donahue, Julian McAuley, and Miller Puckette · 2018
Earlier work this paper cites.
Feature-wise Transformations
Vincent Dumoulin, Ethan Perez, Nathan Schucher, Florian Strub, Harm de Vries, Aaron Courville, and Yoshua Bengio · 2018
Earlier work this paper cites.
Efficient Neural Audio Synthesis
Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Edward Lockhart, Florian Stimberg, Aäron van den Oord, Sander Dieleman, and Koray Kavukcuoglu · 2018
Earlier work this paper cites.

Similar

Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement
Jason Lee, Elman Mansimov, and Kyunghyun Cho · 2018
Cited alongside, same era.
Parallel WaveNet: Fast High-Fidelity Speech Synthesis
Aäron van den Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, George van den Driessche, Edward Lockhart, Luis C. Cobo, Florian Stimberg, Norman Casagrande, Dominik Grewe, Seb Noury, Sander Dieleman, Erich Elsen, Nal Kalchbrenner, Heiga Zen, Alex Graves, Helen King, Tom Walters, Dan Belov, and Demis Hassabis · 2018
Cited alongside, same era.
Deep Energy Estimator Networks
Original
Saeed Saremi, Arash Mehrjou, Bernhard Schölkopf, and Aapo Hyvärinen · 2018
Cited alongside, same era.
Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions
Jonathan Shen, Ruoming Pang, Ron J. Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, RJ Skerrv-Ryan, Rif A. Saurous, Yannis Agiomyrgiannakis, and Yonghui Wu · 2018
Cited alongside, same era.
Parrotron: An End-to-End Speech-to-Speech Conversion Model and its Applications to Hearing-Impaired Speech and Speech Separation
Fadi Biadsy, Ron J. Weiss, Pedro J. Moreno, Dimitri Kanevsky, and Ye Jia · 2019
Cited alongside, same era.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova · 2019
Cited alongside, same era.
GANSynth: Adversarial Neural Audio Synthesis
Jesse Engel, Kumar Krishna Agrawal, Shuo Chen, Ishaan Gulrajani, Chris Donahue, and Adam Roberts · 2019
Cited alongside, same era.
Mask-Predict: Parallel Decoding of Conditional Masked Language Models
Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer · 2019
Cited alongside, same era.
Levenshtein Transformer
Jiatao Gu, Changhan Wang, and Jake Zhao · 2019
Cited alongside, same era.
Direct Speech-to-Speech Translation with a Sequence-to-Sequence Model
Ye Jia, Ron J. Weiss, Fadi Biadsy, Wolfgang Macherey, Melvin Johnson, Zhifeng Chen, and Yonghui Wu · 2019
Cited alongside, same era.
GlotNet—A Raw Waveform Model for the Glottal Excitation in Statistical Parametric Speech Synthesis
Lauri Juvela, Bajibabu Bollepalli, Vassilis Tsiaras, and Paavo Alku · 2019
Cited alongside, same era.
FloWaveNet: A Generative Flow for Raw Audio
Sungwon Kim, Sang-Gil Lee, Jongyoon Song, Jaehyeon Kim, and Sungroh Yoon · 2019
Cited alongside, same era.

Then

Insertion Transformer: Flexible Sequence Generation via Insertion Operations
Mitchell Stern, William Chan, Jamie Kiros, and Jakob Uszkoreit · 2019
Later among the works it cites.
LPCNet: Improving Neural Speech Synthesis through Linear Prediction
Jean-Marc Valin and Jan Skoglund · 2019
Later among the works it cites.
MelNet: A Generative Model for Audio in the Frequency Domain
Original
Sean Vasquez and Mike Lewis · 2019
Later among the works it cites.
Knowledge-and-Data-Driven Amplitude Spectrum Prediction for Hierarchical Neural Vocoders
Original
Yang Ai and Zhen-Hua Ling · 2020
Closest in time.
Location-relative Attention Mechanisms for Robust Long-form Speech Synthesis
Eric Battenberg, RJ Skerry-Ryan, Soroosh Mariooryad, Daisy Stanton, David Kao, Matt Shannon, and Tom Bagby · 2020
Closest in time.
High Fidelity Speech Synthesis with Adversarial Networks
Mikołaj Bińkowski, Jeff Donahue, Sander Dieleman, Aidan Clark, Erich Elsen, Norman Casagrande, Luis C. Cobo, and Karen Simonyan · 2020
Closest in time.
Learning Gradient Fields for Shape Generation
Ruojin Cai, Guandao Yang, Hadar Averbuch-Elor, Zekun Hao, Serge Belongie, Noah Snavely, and Bharath Hariharan · 2020
Closest in time.
Imputer: Sequence Modelling via Imputation and Dynamic Programming
William Chan, Chitwan Saharia, Geoffrey Hinton, Mohammad Norouzi, and Navdeep Jaitly · 2020
Closest in time.
DDSP: Differentiable Digital Signal Processing
Jesse Engel, Lamtharn Hantrakul, Chenjie Gu, and Adam Roberts · 2020
Closest in time.
A Spectral Energy Distance for Parallel Speech Synthesis
Original
Alexey A. Gritsenko, Tim Salimans, Rianne van den Berg, Jasper Snoek, and Nal Kalchbrenner · 2020
Closest in time.
Denoising Diffusion Probabilistic Models
Original
Jonathan Ho, Ajay Jain, and Pieter Abbeel · 2020
Closest in time.
WaveNODE: A Continuous Normalizing Flow for Speech Synthesis
Hyeongju Kim, Hyeonseung Lee, Woo Hyun Kang, Sung Jun Cheon, Byoung Jin Choi, and Nam Soo Kim · 2020
Closest in time.
DiffWave: A Versatile Diffusion Model for Audio Synthesis
Original
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro · 2020
Closest in time.
HooliGAN: Robust, High Quality Neural Vocoding
Original
Ollie McCarthy and Zohaib Ahmed · 2020
Closest in time.
Non-Autoregressive Neural Text-to-Speech
Kainan Peng, Wei Ping, Zhao Song, and Kexin Zhao · 2020
Closest in time.
Non-Autoregressive Machine Translation with Latent Alignments
Original
Chitwan Saharia, William Chan, Saurabh Saxena, and Mohammad Norouzi · 2020
Closest in time.
Improved Techniques for Training Score-Based Generative Models
Original
Yang Song and Stefano Ermon · 2020
Closest in time.
Sliced Score Matching: A Scalable Approach to Density and Score Estimation
Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon · 2020
Closest in time.
Neural Source-Filter Waveform Models for Statistical Parametric Speech Synthesis
Xin Wang, Shinji Takaki, and Junichi Yamagishi · 2020
Closest in time.
WaveFFJORD: FFJORD-Based Vocoder for Statistical Parametric Speech Synthesis
Ning-Qian Wu and Zhen-Hua Ling · 2020
Closest in time.
Parallel WaveGAN: A Fast Waveform Generation Model Based on Generative Adversarial Networks with Multi-Resolution Spectrogram
Ryuichi Yamamoto, Eunwoo Song, and Jae-Min Kim · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…