2018

Neural Voice Cloning with a Few Samples

Arik, Sercan O., Chen, Jitong, Peng, Kainan et al.

Understand

Voice cloning is a highly desired feature for personalized speech interfaces.

Neural network based speech synthesis has been shown to generate high quality speech for a large number of speakers.
In this paper, we introduce a neural voice cloning system that takes a few audio samples as input.
We study two approaches: speaker adaptation and speaker encoding.

Built on

Probabilistic linear discriminant analysis for inferences about identity
S. Prince and J. Elder · 2007
Earlier work this paper cites.
Analysis of speaker adaptation algorithms for hmm-based speech synthesis and a constrained smaplr adaptation algorithm
J. Yamagishi, T. Kobayashi, Y. Nakano, K. Ogata, and J. Isogai · 2009
Earlier work this paper cites.
Spectral mapping using artificial neural networks for voice conversion
S. Desai, A. W. Black, B. Yegnanarayana, and K. Prahallad · 2010
Earlier work this paper cites.
Fast speaker adaptation of hybrid nn/hmm model for speech recognition based on discriminative learning of speaker code
O. Abdel-Hamid and H. Jiang · 2013
Earlier work this paper cites.
One-shot learning by inverting a compositional causal process
B. M. Lake, R. Salakhutdinov, and J. B. Tenenbaum · 2013
Earlier work this paper cites.
Kl-divergence regularized deep neural network adaptation for improved large vocabulary speech recognition
D. Yu, K. Yao, H. Su, G. Li, and F. Seide · 2013
Earlier work this paper cites.
Voice conversion using deep neural networks with layer-wise generative training
L. H. Chen, Z. H. Ling, L. J. Liu, and L. R. Dai · 2014
Earlier work this paper cites.
One-shot learning of generative speech concepts
B. M. Lake, C. ying Lee, J. R. Glass, and J. B. Tenenbaum · 2014
Earlier work this paper cites.
Fast adaptation of deep neural network based on discriminant codes for speech recognition
S. Xue, O. Abdel-Hamid, H. Jiang, L. Dai, and Q. Liu · 2014
Earlier work this paper cites.
A probabilistic interpretation for artificial neural network-based voice conversion
H. T. Hwang, Y. Tsao, H. M. Wang, Y. R. Wang, and S. H. Chen · 2015
Earlier work this paper cites.
Human-level concept learning through probabilistic program induction
B. M. Lake, R. Salakhutdinov, and J. B. Tenenbaum · 2015
Earlier work this paper cites.
Modeling speaker variability using long short-term memory networks for speech recognition
X. Li and X. Wu · 2015
Earlier work this paper cites.

Similar

On speaker adaptation of long short-term memory recurrent neural networks
Y. Miao and F. Metze · 2015
Cited alongside, same era.
Speaker adaptive training of deep neural network acoustic models using i-vectors
Y. Miao, H. Zhang, and F. Metze · 2015
Cited alongside, same era.
Librispeech: an ASR corpus based on public domain audio books
V. Panayotov, G. Chen, D. Povey, and S. Khudanpur · 2015
Cited alongside, same era.
A study of speaker adaptation for dnn-based speech synthesis
Z. Wu, P. Swietojanski, C. Veaux, S. Renals, and S. King · 2015
Cited alongside, same era.
Voice morphing that improves tts quality using an optimal dynamic frequency warping-and-weighting transform
Y. Agiomyrgiannakis and Z. Roupakia · 2016
Cited alongside, same era.
Deep speech 2: End-to-end speech recognition in english and mandarin
D. Amodei, S. Ananthanarayanan, R. Anubhai, J. Bai, E. Battenberg, C. Case, J. Casper, B. Catanzaro, Q. Cheng, G. Chen, et al · 2016
Cited alongside, same era.
Exploring the limits of language modeling
Original
R. Jozefowicz, O. Vinyals, M. Schuster, N. Shazeer, and Y. Wu · 2016
Cited alongside, same era.
Samplernn: An unconditional end-to-end neural audio generation model
Original
S. Mehri, K. Kumar, I. Gulrajani, R. Kumar, S. Jain, J. Sotelo, A. Courville, and Y. Bengio · 2016
Cited alongside, same era.
One-shot generalization in deep generative models
D. Rezende, Shakir, I. Danihelka, K. Gregor, and D. Wierstra · 2016
Cited alongside, same era.
Deep neural network-based speaker embeddings for end-to-end speaker verification
D. Snyder, P. Ghahremani, D. Povey, D. Garcia-Romero, Y. Carmiel, and S. Khudanpur · 2016
Cited alongside, same era.
Analysis of the voice conversion challenge 2016 evaluation results
M. Wester, Z. Wu, and J. Yamagishi · 2016
Cited alongside, same era.
Locally linear embedding for exemplar-based spectral conversion
Y.-C. Wu, H.-T. Hwang, C.-C. Hsu, Y. Tsao, and H.-m. Wang · 2016
Cited alongside, same era.

Then

Multi-content gan for few-shot font style transfer
Original
S. Azadi, M. Fisher, V. Kim, Z. Wang, E. Shechtman, and T. Darrell · 2017
Later among the works it cites.
Embedding-based speaker adaptive training of deep neural networks
Original
X. Cui, V. Goel, and G. Saon · 2017
Later among the works it cites.
Progressive growing of gans for improved quality, stability, and variation
Original
T. Karras, T. Aila, S. Laine, and J. Lehtinen · 2017
Later among the works it cites.
Few-shot autoregressive density estimation: Towards learning to learn distributions
S. E. Reed, Y. Chen, T. Paine, A. van den Oord, S. M. A. Eslami, D. J. Rezende, O. Vinyals, and N. de Freitas · 2017
Later among the works it cites.
Natural tts synthesis by conditioning wavenet on mel spectrogram predictions
Original
J. Shen, R. Pang, R. J. Weiss, M. Schuster, N. Jaitly, Z. Yang, Z. Chen, Y. Zhang, Y. Wang, R. Skerry-Ryan, et al · 2017
Later among the works it cites.
Char2wav: End-to-end speech synthesis
J. Sotelo, S. Mehri, K. Kumar, J. F. Santos, K. Kastner, A. Courville, and Y. Bengio · 2017
Later among the works it cites.
Attention is all you need
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin · 2017
Later among the works it cites.
Cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit, 2017
C. Veaux, J. Yamagishi, and K. e. a. MacDonald · 2017
Later among the works it cites.
Tacotron: A fully end-to-end text-to-speech synthesis model
Original
Y. Wang, R. J. Skerry-Ryan, D. Stanton, Y. Wu, R. J. Weiss, N. Jaitly, Z. Yang, Y. Xiao, Z. Chen, S. Bengio, Q. V. Le, Y. Agiomyrgiannakis, R. Clark, and R. A. Saurous · 2017
Later among the works it cites.
Deep Voice 3: Scaling text-to-speech with convolutional sequence learning
W. Ping, K. Peng, A. Gibiansky, S. Arik, A. Kannan, S. Narang, J. Raiman, and J. Miller · 2018
Closest in time.
Voiceloop: Voice fitting and synthesis via a phonological loop
Y. Taigman, L. Wolf, A. Polyak, and E. Nachmani · 2018
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…