2021

Expressive Neural Voice Cloning

Neekhara, Paarth, Hussain, Shehzeen, Dubnov, Shlomo et al.

Understand

Voice cloning is the task of learning to synthesize the voice of an unseen speaker from a few samples.

While current voice cloning methods achieve promising results in Text-to-Speech (TTS) synthesis for a new voice, these approaches lack the ability to control the expressiveness of synthesized audio.
In this work, we propose a controllable voice cloning method that allows fine-grained control over various style aspects of the synthesized speech for an unseen speaker.
We achieve this by explicitly conditioning the speech synthesis model on a speaker encoding, pitch contour and latent style tokens during training.

Built on

Signal estimation from modified short-time Fourier transform
Daniel W. Griffin, Jae, S. Lim, and Senior Member · 1984
Earlier work this paper cites.
Yin, a fundamental frequency estimator for speech and music
Alain De Cheveigné and Hideki Kawahara · 2002
Earlier work this paper cites.
A method for fundamental frequency estimation and voicing decision: Application to infant utterances recorded in real acoustical environments
Tomohiro Nakatani, Shigeaki Amano, Toshio Irino, Kentaro Ishizuka, and Tadahisa Kondo · 2008
Earlier work this paper cites.
Reducing f0 frame error of f0 tracking algorithms under noisy conditions with an unvoiced/voiced classification frontend
Wei Chu and Abeer Alwan · 2009
Earlier work this paper cites.
The blizzard challenge 2013
Simon J. King and Vasilis Karaiskos · 2013
Earlier work this paper cites.
Adam: A method for stochastic optimization
Diederik P Kingma and Jimmy Ba · 2015
Earlier work this paper cites.
Librispeech: an asr corpus based on public domain audio books
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur · 2015
Earlier work this paper cites.
WaveNet: A generative model for raw audio
Original
Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu · 2016
Earlier work this paper cites.
Deep voice 2: Multi-speaker neural text-to-speech
Andrew Gibiansky, Sercan Arik, Gregory Diamos, John Miller, Kainan Peng, Wei Ping, Jonathan Raiman, and Yanqi Zhou · 2017
Earlier work this paper cites.
The lj speech dataset
Keith Ito · 2017
Earlier work this paper cites.
Cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit
Christophe Veaux, Junichi Yamagishi, and Kirsten Macdonald · 2017
Earlier work this paper cites.

Similar

Generalized end-to-end loss for speaker verification
Original
Li Wan, Quan Wang, Alan Papir, and Ignacio Lopez Moreno · 2017
Cited alongside, same era.
Tacotron: Towards end-to-end speech synthesis
Yuxuan Wang, RJ Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, et al · 2017
Cited alongside, same era.
Neural voice cloning with a few samples
Sercan Arik, Jitong Chen, Kainan Peng, Wei Ping, and Yanqi Zhou · 2018
Cited alongside, same era.
Voxceleb2: Deep speaker recognition
J. S. Chung, A. Nagrani, and A. Zisserman · 2018
Cited alongside, same era.
Transfer learning from speaker verification to multispeaker text-to-speech synthesis
Ye Jia, Yu Zhang, Ron Weiss, Quan Wang, Jonathan Shen, Fei Ren, zhifeng Chen, Patrick Nguyen, Ruoming Pang, Ignacio Lopez Moreno, and Yonghui Wu · 2018
Cited alongside, same era.
Deep Voice 3: Scaling text-to-speech with convolutional sequence learning
Wei Ping, Kainan Peng, Andrew Gibiansky, Sercan O Arik, Ajay Kannan, Sharan Narang, Jonathan Raiman, and John Miller · 2018
Cited alongside, same era.
Deep voice 3: 2000-speaker neural text-to-speech
Wei Ping, Kainan Peng, Andrew Gibiansky, Sercan Ö. Arik, Ajay Kannan, Sharan Narang, Jonathan Raiman, and John L. Miller · 2018
Cited alongside, same era.
WaveGlow: A flow-based generative network for speech synthesis
Ryan Prenger, Rafael Valle, and Bryan Catanzaro · 2018
Cited alongside, same era.
Natural TTS synthesis by conditioning WaveNet on mel spectrogram predictions
Jonathan Shen, Ruoming Pang, Ron J Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, et al · 2018
Cited alongside, same era.
Towards end-to-end prosody transfer for expressive speech synthesis with tacotron
Original
R. J. Skerry-Ryan, Eric Battenberg, Ying Xiao, Yuxuan Wang, Daisy Stanton, Joel Shor, Ron J. Weiss, Rob Clark, and Rif A. Saurous · 2018
Cited alongside, same era.
Predicting expressive speaking style from text in end-to-end speech synthesis
Original
Daisy Stanton, Yuxuan Wang, and R. J. Skerry-Ryan · 2018
Cited alongside, same era.

Then

Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis
Original
Yuxuan Wang, Daisy Stanton, Yu Zhang, RJ Skerry-Ryan, Eric Battenberg, Joel Shor, Ying Xiao, Fei Ren, Ye Jia, and Rif A. Saurous · 2018
Later among the works it cites.
Cross-lingual, multi-speaker text-to-speech synthesis using neural speaker embedding
Mengnan Chen, Minchuan Chen, Shuang Liang, Jun Ma, Lei Chen, Shaojun Wang, and Jing Xiao · 2019
Later among the works it cites.
Master thesis : Automatic multispeaker voice cloning
Gilles Louppe · 2019
Later among the works it cites.
Resemblyzer - https://github.com/resemble-ai/Resemblyzer/
Gilles Louppe · 2019
Later among the works it cites.
Voxceleb: Large-scale speaker verification in the wild
Arsha Nagrani, Joon Son Chung, Weidi Xie, and Andrew Zisserman · 2019
Later among the works it cites.
Expediting TTS synthesis with adversarial vocoding
Paarth Neekhara, Chris Donahue, Miller Puckette, Shlomo Dubnov, and Julian McAuley · 2019
Later among the works it cites.
LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech
Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J. Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu · 2019
Later among the works it cites.
Zero-shot multi-speaker text-to-speech with state-of-the-art neural speaker embeddings
E. Cooper, C. Lai, Y. Yasuda, F. Fang, X. Wang, N. Chen, and J. Yamagishi · 2020
Later among the works it cites.
Using personalized speech synthesis and neural language generator for rapid speaker adaptation
Y. Huang, L. He, W. Wei, W. Gale, J. Li, and Y. Gong · 2020
Later among the works it cites.
Mellotron: Multispeaker expressive voice synthesis by conditioning on rhythm, pitch and global style tokens
Rafael Valle, Jason Li, Ryan Prenger, and Bryan Catanzaro · 2020
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…