2022

VarietySound: Timbre-Controllable Video to Sound Generation via Unsupervised Information Disentanglement

Cui, Chenye, Ren, Yi, Liu, Jinglin et al.

Understand

Video to sound generation aims to generate realistic and natural sound given a video input.

However, previous video-to-sound generation methods can only generate a random or average timbre without any controls or specializations of the generated sound timbre, leading to the problem that people cannot obtain the desired timbre under these methods sometimes.
In this paper, we pose the task of generating sound with a specific timbre given a video input and a reference audio sample.
To solve this task, we disentangle each target sound audio into three components: temporal information, acoustic information, and background information.

Built on

High-quality speech synthesis using super-resolution mel-spectrogram
Original
Sheng, L.; Huang, D.-Y.; and Pavlovskiy, E. N. 2019 · 1912
Earlier work this paper cites.
Mel-cepstral distance measure for objective speech quality assessment
Kubichek, R. 1993 · 1993
Earlier work this paper cites.
Generative adversarial nets
Goodfellow, I.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; and Bengio, Y. 2014 · 2014
Earlier work this paper cites.
Batch normalization: Accelerating deep network training by reducing internal covariate shift
Ioffe, S.; and Szegedy, C. 2015 · 2015
Earlier work this paper cites.
Soundnet: Learning sound representations from unlabeled video
Aytar, Y.; Vondrick, C.; and Torralba, A. 2016 · 2016
Earlier work this paper cites.
Wavenet: A generative model for raw audio
Original
Oord, A. v. d.; Dieleman, S.; Zen, H.; Simonyan, K.; Vinyals, O.; Graves, A.; Kalchbrenner, N.; Senior, A.; and Kavukcuoglu, K. 2016 · 2016
Earlier work this paper cites.
Aenet: Learning deep audio features for video analysis
Takahashi, N.; Gygli, M.; and Van Gool, L. 2017 · 2017
Earlier work this paper cites.
Attention is all you need
Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017 · 2017
Earlier work this paper cites.
Visually indicated sound generation by perceptually optimized classification
Chen, K.; Zhang, C.; Fang, C.; Wang, Z.; Bui, T.; and Nevatia, R. 2018 · 2018
Earlier work this paper cites.
Adversarial audio synthesis
Original
Donahue, C.; McAuley, J.; and Puckette, M. 2018 · 2018
Earlier work this paper cites.
Learning to decompose and disentangle representations for video prediction
Hsieh, J.-T.; Liu, B.; Huang, D.-A.; Fei-Fei, L. F.; and Niebles, J. C. 2018 · 2018
Earlier work this paper cites.
Decoupled Weight Decay Regularization
Loshchilov, I.; and Hutter, F. 2018 · 2018
Earlier work this paper cites.

Similar

Visual to sound: Generating natural sound for videos in the wild
Zhou, Y.; Wang, Z.; Fang, C.; Bui, T.; and Berg, T. L. 2018 · 2018
Cited alongside, same era.
Large scale adversarial representation learning
Donahue, J.; and Simonyan, K. 2019 · 2019
Cited alongside, same era.
A style-based generator architecture for generative adversarial networks
Karras, T.; Laine, S.; and Aila, T. 2019 · 2019
Cited alongside, same era.
Melgan: Generative adversarial networks for conditional waveform synthesis
Kumar, K.; Kumar, R.; de Boissiere, T.; Gestin, L.; Teoh, W. Z.; Sotelo, J.; de Brébisson, A.; Bengio, Y.; and Courville, A. C. 2019 · 2019
Cited alongside, same era.
Disentangling disentanglement in variational autoencoders
Mathieu, E.; Rainforth, T.; Siddharth, N.; and Teh, Y. W. 2019 · 2019
Cited alongside, same era.
Autovc: Zero-shot voice style transfer with only autoencoder loss
Qian, K.; Zhang, Y.; Chang, S.; Yang, X.; and Hasegawa-Johnson, M. 2019 · 2019
Cited alongside, same era.
Generating visually aligned sound from videos
Chen, P.; Zhang, Y.; Tan, M.; Xiao, H.; Huang, D.; and Gan, C. 2020 · 2020
Cited alongside, same era.
HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis
Kong, J.; Kim, J.; and Bae, J. 2020 · 2020
Cited alongside, same era.
Unsupervised speech decomposition via triple information bottleneck
Qian, K.; Zhang, Y.; Chang, S.; Hasegawa-Johnson, M.; and Cox, D. 2020 · 2020
Cited alongside, same era.
FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
Ren, Y.; Hu, C.; Tan, X.; Qin, T.; Zhao, S.; Zhao, Z.; and Liu, T.-Y. 2020 · 2020
Cited alongside, same era.
Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram
Yamamoto, R.; Song, E.; and Kim, J.-M. 2020 · 2020
Cited alongside, same era.
Singgan: Generative adversarial network for high-fidelity singing voice generation
Huang, R.; Cui, C.; Chen, F.; Ren, Y.; Liu, J.; Zhao, Z.; Huai, B.; and Wang, Z. 2022a
Cited in the paper.

Then

Disentanglement Learning for Variational Autoencoders Applied to Audio-Visual Speech Enhancement
Carbajal, G.; Richter, J.; and Gerkmann, T. 2021 · 2021
Later among the works it cites.
EMOVIE: A Mandarin Emotion Speech Dataset with a Simple Emotional Text-to-Speech Model
Original
Cui, C.; Ren, Y.; Liu, J.; Chen, F.; Huang, R.; Lei, M.; and Zhao, Z. 2021 · 2021
Later among the works it cites.
FoleyGAN: Visually Guided Generative Adversarial Network-Based Synchronous Sound Generation in Silent Videos
Original
Ghose, S.; and Prevost, J. J. 2021 · 2021
Later among the works it cites.
Multi-singer: Fast multi-singer singing voice vocoder with a large-scale corpus
Huang, R.; Chen, F.; Ren, Y.; Liu, J.; Cui, C.; and Zhao, Z. 2021 · 2021
Later among the works it cites.
Taming Visually Guided Sound Generation
Iashin, V.; and Rahtu, E. 2021 · 2021
Later among the works it cites.
Bilateral denoising diffusion models
Original
Lam, M. W.; Wang, J.; Huang, R.; Su, D.; and Yu, D. 2021 · 2021
Later among the works it cites.
Multi-SpectroGAN: High-Diversity and High-Fidelity Spectrogram Generation with Adversarial Style Combination for Speech Synthesis
Lee, S.-H.; Yoon, H.-W.; Noh, H.-R.; Kim, J.-H.; and Lee, S.-W. 2021 · 2021
Later among the works it cites.
GANSpeech: Adversarial Training for High-Fidelity Multi-Speaker Speech Synthesis
Original
Yang, J.; Bae, J.-S.; Bak, T.; Kim, Y.; and Cho, H.-Y. 2021 · 2021
Later among the works it cites.
A Practical Data-Free Approach to One-shot Federated Learning with Heterogeneity
Original
Zhang, J.; Chen, C.; Li, B.; Lyu, L.; Wu, S.; Xu, J.; Ding, S.; and Wu, C. 2021 · 2021
Later among the works it cites.
Meta-RangeSeg: LiDAR Sequence Semantic Segmentation Using Multiple Feature Aggregation
Wang, S.; Zhu, J.; and Zhang, R. 2022 · 2022
Closest in time.
SyntaSpeech: Syntax-Aware Generative Adversarial Text-to-Speech
Original
Ye, Z.; Zhao, Z.; Ren, Y.; and Wu, F. 2022 · 2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…