2022

Universal Speech Enhancement with Score-based Diffusion

Serrà, Joan, Pascual, Santiago, Pons, Jordi et al.

Understand

Removing background noise from speech audio has been the subject of considerable effort, especially in recent years due to the rise of virtual communication and amateur recordings.

Yet background noise is not the only unpleasant disturbance that can prevent intelligibility: reverb, clipping, codec artifacts, problematic equalization, limited bandwidth, or inconsistent loudness are equally disturbing and ubiquitous.
In this work, we propose to consider the task of speech enhancement as a holistic endeavor, and present a universal speech enhancement system that tackles 55 different distortions at the same time.
Our approach consists of a generative model that employs score-based diffusion, together with a multi-resolution conditioning network that performs enhancement with mixture density networks.

Built on

Interpolation and extrapolation of stationary random sequences
A. Kolmogorov · 1941
Earlier work this paper cites.
Interpolation, extrapolation, and smoothing of stationary time series
N. Wiener · 1949
Earlier work this paper cites.
Reverse-time diffusion equation models
B. D. O. Anderson · 1982
Earlier work this paper cites.
Mixture density networks
C. M. Bishop · 1994
Earlier work this paper cites.
Perceptual evaluation of speech quality (PESQ) – A new method for speech quality assessment of telephone networks and codecs
A. W. Rix, J. G. Beerends, M. P. Hollier, and A. P. Hekstra · 2001
Earlier work this paper cites.
Estimation of non-normalized statistical models by score matching
A. Hyvärinen · 2005
Earlier work this paper cites.
A short-time objective intelligibility measure for time-frequency weighted noisy speech
C. H. Taal, R. C. Hendriks, R. Heusdens, and J. Jensen · 2010
Earlier work this paper cites.
A connection between score matching and denoising autoencoders
P. Vincent · 2011
Earlier work this paper cites.
Speech enhancement
P. C. Loizou · 2013
Earlier work this paper cites.
Speech enhancement based on deep denoising autoencoder
X. Lu, Y. Tsao, S. Matsuda, and C. Hori · 2013
Earlier work this paper cites.
DEMAND: a collection of multi-channel recordings of acoustic noise in diverse environments (1.0)
J. Thiemann, N. Ito, and E. Vincent · 2013
Earlier work this paper cites.
Repeated Harvard sentence prompts corpus version 0.5
G. E. Henter, T. Merritt, M. Shannon, C. Mayo, and S. King · 2014
Earlier work this paper cites.
Deep unsupervised learning using nonequilibrium thermodynamics
J. Sohl-Dickstein, E. A. Weiss, N. Maheswaranathan, and S. Ganguli · 2015
Earlier work this paper cites.
SEGAN: speech enhancement generative adversarial network
S. Pascual, A. Bonafonte, and J. Serrà · 2017
Earlier work this paper cites.
Noisy speech database for training speech enhancement algorithms and TTS models
C. Valentini-Botinhao · 2017
Earlier work this paper cites.
Crepe: a convolutional representation for pitch estimation
J. W. Kim, J. Salamon, P. Li, and J. P. Bello · 2018
Earlier work this paper cites.
FiLM: visual reasoning with a general conditioning layer
E. Perez, F. Strub, H. de Vries, V. Dumoulin, and A. Courville · 2018
Earlier work this paper cites.
A WaveNet for speech denoising
D. Rethage, J. Pons, and X. Serra · 2018
Earlier work this paper cites.
Learning sound event classifiers from web audio with noisy labels
E. Fonseca, M. Plakal, D. P. W. E. Ellis, F. Font, X. Favory, and X. Serra · 2019
Earlier work this paper cites.
Declipping speech using deep filtering
W. Mack and E. A. P. Habets · 2019
Earlier work this paper cites.
Parametric resynthesis with neural vocoders
S. Maiti and M. I. Mandel · 2019
Earlier work this paper cites.

Similar

Towards generalized speech enhancement with generative adversarial networks
S. Pascual, J. Serrà, and A. Bonafonte · 2019
Cited alongside, same era.
PyTorch: an imperative style, high-performance deep learning library
A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala · 2019
Cited alongside, same era.
Generative modeling by estimating gradients of the data distribution
Y. Song and S. Ermon · 2019
Cited alongside, same era.
Perceptually-motivated environment-specific speech enhancement
J. Su, A. Finkelstein, and Z. Jin · 2019
Cited alongside, same era.
Applied stochastic differential equations , volume 10
S. Särkkä and A. Solin · 2019
Cited alongside, same era.
CSTR VCTK Corpus: English Multi-speaker Corpus for CSTR Voice cloning toolkit (version 0.92)
Y. Yamagishi, C. Veaux, and K. MacDonald · 2019
Cited alongside, same era.
Real time speech enhancement in the waveform domain
A. Défossez, G. Synnaeve, and Y. Adi · 2020
Cited alongside, same era.
Source coding of audio signals with a generative model
R. Fejgin, J. Klejsa, L. Villemoes, and C. Zhou · 2020
Cited alongside, same era.
Denoising diffusion probabilistic models
J. Ho, A. Jain, and P. Abbeel · 2020
Cited alongside, same era.
PoCoNet: better speech enhancement with frequency-positional embeddings, semi-supervised conversational data, and biased loss
U. Isik, R. Giri, N. Phansalkar, J.-M. Valin, K. Helwani, and A. Krishnaswamy · 2020
Cited alongside, same era.
Speaker independence of neural vocoders and their effect on parametric resynthesis speech enhancement
S. Maiti and M. I. Mandel · 2020
Cited alongside, same era.
Improving GANs for speech enhancement
H. Phan, I. V. McLoughlin, L. Pham, O. Y. Chén, P. Koch, M. de Vos, and A. Mertins · 2020
Cited alongside, same era.

Then

Perceptual loss based speech denoising with an ensemble of audio pattern recognition and self-supervised models
S. Kataria, J. Villalba, and N. Dehak · 2021
Later among the works it cites.
SE-Conformer: time-domain speech enhancement using conformer
E. Kim and H. Seo · 2021
Later among the works it cites.
DiffWave: a versatile diffusion model for audio synthesis
Z. Kong, W. Ping, J. Huang, K. Zhao, and B. Catanzaro · 2021
Later among the works it cites.
NU-Wave: a diffusion probabilistic model for neural audio upsampling
Original
J. Lee and S. Han · 2021
Later among the works it cites.
VoiceFixer: toward general speech restoration with neural vocoder
H. Liu, Q. Kong, Q. Tian, Y. Zhao, D. Wang, C. Huang, and Y. Wang · 2021
Later among the works it cites.
A study on speech enhancement based on diffusion probabilistic model
Y.-J. Lu, Y. Tsao, and S. Watanabe · 2021
Later among the works it cites.
DCCRN+: channel-wise subband DCCRN with SNR estimation for speech enhancement
S. Lv, Y. Hu, S. Zhang, and L. Xie · 2021
Later among the works it cites.
Cascaded time + time-frequency UNet for speech enhancement: jointly addressing clipping, codec distortions, and gaps
A. A. Nair and K. Koishida · 2021
Later among the works it cites.
High fidelity speech regeneration with application to speech enhancement
A. Polyak, L. Wolf, Y. Adi, O. Kabeli, and Y. Taigman · 2021
Later among the works it cites.
Upsampling artifacts in neural audio synthesis
J. Pons, S. Pascual, G. Cengarle, and J. Serrà · 2021
Later among the works it cites.
Grad-TTS: a diffusion probabilistic model for text-to-speech
Original
V. Popov, I. Vovk, V. Gogoryan, T. Sadekova, and M. Kudinov · 2021
Later among the works it cites.
CRASH: raw audio score-based generative modeling for controllable high-resolution drum sound synthesis
S. Rouard and G. Hadjeres · 2021
Later among the works it cites.
Score-based generative modeling through stochastic differential equations
Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole · 2021
Later among the works it cites.
A flow-based neural network for time domain speech enhancement
M. Strauss and B. Edler · 2021
Later among the works it cites.
HiFi-GAN-2: studio-quality speech enhancement via generative adversarial networks conditioned on acoustic features
J. Su, Z. Jin, and A. Finkelstein · 2021
Later among the works it cites.
A two-stage complex network using cycle-consistent generative adversarial networks for speech enhancement
G. Yu, Y. Wang, H. Wang, Q. Zhang, and C. Zheng · 2021
Later among the works it cites.
Interactive speech and noise modeling for speech enhancement
C. Zheng, X. Peng, Y. Zhang, S. Srinivasan, and Y. Lu · 2021
Later among the works it cites.
Conditional diffusion probabilistic model for speech enhancement
Y.-J. Lu, Z.-Q. Wang, S. Watanabe, A. Richard, C. Yu, and Y. Tsao · 2022
Closest in time.
Phase sensitive masking-based single channel speech enhancement using conditional generative adversarial network
S. Routray and Q. Mao · 2022
Closest in time.
Speech enhancement with score-based generative models in the complex STFT domain
S. Welker, J. Richter, and T. Gerkmann · 2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…