2018

Conditional End-to-End Audio Transforms

Haque, Albert, Guo, Michelle, Verma, Prateek

Understand

We present an end-to-end method for transforming audio from one style to another.

For the case of speech, by conditioning on speaker identities, we can train a single model to transform words spoken by multiple people into multiple target voices.
For the case of music, we can specify musical instruments and achieve the same result.
Architecturally, our method is a fully-differentiable sequence-to-sequence model based on convolutional and hierarchical recurrent neural networks.

Built on

S. Nawab, T. Quatieri, and J. Lim, “Signal reconstruction from short-time fourier transform magnitude,”
1983
Earlier work this paper cites.
R. J. Williams and D. Zipser, “A learning algorithm for continually running fully recurrent neural networks,”
1989
Earlier work this paper cites.
H. Bourlard and N. Morgan, “A continuous speech recognition system embedding mlp into hmm,” in
1990
Earlier work this paper cites.
J. G. Wilpon, L. R. Rabiner, C.-H. Lee, and E. Goldman, “Automatic recognition of keywords in unconstrained speech using hidden markov models,”
1990
Earlier work this paper cites.
M. Abe, S. Nakamura, K. Shikano, and H. Kuwabara, “Voice conversion through vector quantization,”
1990
Earlier work this paper cites.
P. Dutilleux, “An implementation of the “algorithme à trous” to compute the wavelet transform,” in
1990
Earlier work this paper cites.
J. S. Garofolo, L. F. Lamel, W. M. Fisher, J. G. Fiscus, and D. S. Pallett, “Darpa timit acoustic-phonetic continous speech corpus cd-rom. nist speech disc 1-1.1,”
1993
Earlier work this paper cites.
Y. Bengio, P. Simard, and P. Frasconi, “Learning long-term dependencies with gradient descent is difficult,”
1994
Earlier work this paper cites.
A. Kain and M. W. Macon, “Spectral voice conversion for text-to-speech synthesis,” in
1998
Earlier work this paper cites.
Y. Stylianou, O. Cappé, and E. Moulines, “Continuous probabilistic transform for voice conversion,”
1998
Earlier work this paper cites.
S. S. Haykin
2001
Earlier work this paper cites.
T. Toda, A. W. Black, and K. Tokuda, “Voice conversion based on maximum-likelihood estimation of spectral parameter trajectory,”
2007
Earlier work this paper cites.
L. v. d. Maaten and G. Hinton, “Visualizing data using t-sne,”
2008
Earlier work this paper cites.
B. Gold, N. Morgan, and D. Ellis,
2011
Earlier work this paper cites.
W. Y. Zou, R. Socher, D. M. Cer, and C. D. Manning, “Bilingual word embeddings for phrase-based machine translation.” in
2013
Earlier work this paper cites.
T. Mikolov, K. Chen, G. Corrado, and J. Dean, “Efficient estimation of word representations in vector space,”
2013
Earlier work this paper cites.

Similar

J. Pennington, R. Socher, and C. D. Manning, “Glove: Global vectors for word representation.” in
2014
Cited alongside, same era.
S. H. Mohammadi and A. Kain, “Voice conversion using deep neural networks with speaker-independent pre-training,” in
2014
Cited alongside, same era.
I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence learning with neural networks,” in
2014
Cited alongside, same era.
J. Koutnik, K. Greff, F. Gomez, and J. Schmidhuber, “A clockwork rnn,”
2014
Cited alongside, same era.
D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,”
2014
Cited alongside, same era.
M.-T. Luong, H. Pham, and C. D. Manning, “Effective approaches to attention-based neural machine translation,”
2015
Cited alongside, same era.
L. Gatys, A. S. Ecker, and M. Bethge, “Texture synthesis using convolutional neural networks,” in
2015
Cited alongside, same era.
T. N. Sainath, R. J. Weiss, A. Senior, K. W. Wilson, and O. Vinyals, “Learning the speech front-end with raw waveform cldnns,” in
2015
Cited alongside, same era.
F. Yu and V. Koltun, “Multi-scale context aggregation by dilated convolutions,”
2015
Cited alongside, same era.
J. K. Chorowski, D. Bahdanau, D. Serdyuk, K. Cho, and Y. Bengio, “Attention-based models for speech recognition,” in
2015
Cited alongside, same era.
D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,”
2015
Cited alongside, same era.
A. v. d. Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu, “Wavenet: A generative model for raw audio,”
2016
Cited alongside, same era.

Then

W. Wang, S. Xu, and B. Xu, “First step towards end-to-end parametric tts synthesis: Generating spectral parameters with neural attention,” in
2016
Later among the works it cites.
C.-C. Chiu, T. N. Sainath, Y. Wu, R. Prabhavalkar, P. Nguyen, Z. Chen, A. Kannan, R. J. Weiss, K. Rao, K. Gonina
2017
Later among the works it cites.
Y. Wang, R. Skerry-Ryan, D. Stanton, Y. Wu, R. J. Weiss, N. Jaitly, Z. Yang, Y. Xiao, Z. Chen, S. Bengio
2017
Later among the works it cites.
T. Kinnunen, L. Juvela, P. Alku, and J. Yamagishi, “Non-parallel voice conversion using i-vector plda: Towards unifying speaker verification and transformation,” in
2017
Later among the works it cites.
T. Kaneko, H. Kameoka, K. Hiramatsu, and K. Kashino, “Sequence-to-sequence voice conversion with similarity metric learned using generative adversarial networks,” in
2017
Later among the works it cites.
J. Chorowski, R. J. Weiss, R. A. Saurous, and S. Bengio, “On using backpropagation for speech texture generation and voice conversion,”
2017
Later among the works it cites.
A. van den Oord, O. Vinyals
2017
Later among the works it cites.
J. Sotelo, S. Mehri, K. Kumar, J. F. Santos, K. Kastner, A. Courville, and Y. Bengio, “Char2wav: End-to-end speech synthesis,”
2017
Later among the works it cites.
S. O. Arik, M. Chrzanowski, A. Coates, G. Diamos, A. Gibiansky, Y. Kang, X. Li, J. Miller, J. Raiman, S. Sengupta
2017
Later among the works it cites.
J. Engel, C. Resnick, A. Roberts, S. Dieleman, D. Eck, K. Simonyan, and M. Norouzi, “Neural audio synthesis of musical notes with wavenet autoencoders,” 2017
2017
Later among the works it cites.
J. F. Gemmeke, D. P. W. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, “Audio set: An ontology and human-labeled dataset for audio events,” in
2017
Later among the works it cites.
J. Shen, R. Pang, R. J. Weiss, M. Schuster, N. Jaitly, Z. Yang, Z. Chen, Y. Zhang, Y. Wang, R. Skerry-Ryan, R. A. Saurous, Y. Agiomyrgiannakis, and Y. Wu, “Natural tts synthesis by conditioning wavenet on mel spectrogram predictions,” in
2018
Closest in time.
P. Verma and J. O. Smith, “Neural style transfer for audio spectograms,”
2018
Closest in time.
Y. Wang, D. Stanton, Y. Zhang, R. Skerry-Ryan, E. Battenberg, J. Shor, Y. Xiao, F. Ren, Y. Jia, and R. A. Saurous, “Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis,”
2018
Closest in time.
N. Mor, L. Wolf, A. Polyak, and Y. Taigman, “A universal music translation network,”
2018
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…