2020

Jukebox: A Generative Model for Music

Dhariwal, Prafulla, Jun, Heewoo, Payne, Christine et al.

Understand

We introduce Jukebox, a model that generates music with singing in the raw audio domain.

We tackle the long context of raw audio using a multi-scale VQ-VAE to compress it to discrete codes, and modeling those using autoregressive Transformers.
We show that the combined model at scale can generate high-fidelity and diverse songs with coherence up to multiple minutes.
We can condition on artist and genre to steer the musical and vocal style, and on unaligned lyrics to make the singing more controllable.

Built on

Musical composition with a high speed digital computer
Hiller Jr, L. A. and Isaacson, L. M · 1957
Earlier work this paper cites.
Music and computer composition
Moorer, J. A · 1972
Earlier work this paper cites.
Software for a cascade/parallel formant synthesizer
Klatt, D. H · 1980
Earlier work this paper cites.
Nonlinear maps as generators of musical design
Pressing, J · 1988
Earlier work this paper cites.
The musical universe of cellular automata
Beyls, P · 1989
Earlier work this paper cites.
Unit selection in a concatenative speech synthesis system using a large speech database
Hunt, A. J. and Black, A. W · 1996
Earlier work this paper cites.
Creating music by listening
Jehan, T · 2005
Earlier work this paper cites.
An HMM-based singing voice synthesis system
Saino, K., Zen, H., Nankaku, Y., Lee, A., and Tokuda, K · 2006
Earlier work this paper cites.
Synthesis of the singing voice by performance sampling and spectral models
Bonada, J. and Serra, X · 2007
Earlier work this paper cites.
Deep boltzmann machines
Salakhutdinov, R. and Hinton, G · 2009
Earlier work this paper cites.
Review: Statistical parametric speech synthesis
Zen, H., Tokuda, K., and Black, A. W · 2009
Earlier work this paper cites.
Recent development of the HMM-based singing voice synthesis system – Sinsy
Oura, K., Mase, A., Yamada, T., Muto, S., Nankaku, Y., and Tokuda, K · 2010
Earlier work this paper cites.
Constraint programming systems for modeling music theories and composition
Anders, T. and Miranda, E. R · 2011
Earlier work this paper cites.
Front-end factor analysis for speaker verification
Dehak, N., Kenny, P. J., Dehak, R., Dumouchel, P., and Ouellet, P · 2011
Earlier work this paper cites.
Signal reconstruction from stft magnitude: A state of the art
Sturmel, N. and Daudet, L · 2011
Earlier work this paper cites.
Compact language detector 2
Sites, D · 2013
Earlier work this paper cites.
Generative adversarial nets
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y · 2014
Earlier work this paper cites.
Auto-encoding variational bayes
Kingma, D. P. and Welling, M · 2014
Earlier work this paper cites.
Stochastic backpropagation and approximate inference in deep generative models
Rezende, D. J., Mohamed, S., and Wierstra, D · 2014
Earlier work this paper cites.
Sequence to sequence learning with neural networks
Sutskever, I., Vinyals, O., and Le, Q. V · 2014
Earlier work this paper cites.
NICE: Non-linear independent components estimation
Dinh, L., Krueger, D., and Bengio, Y · 2015
Earlier work this paper cites.
Variational inference with normalizing flows
Rezende, D. and Mohamed, S · 2015
Earlier work this paper cites.
Layer normalization
Original
Ba, J. L., Kiros, J. R., and Hinton, G. E · 2016
Earlier work this paper cites.
NIPS 2016 tutorial: Generative adversarial networks
Goodfellow, I · 2016
Earlier work this paper cites.
Improved variational inference with inverse autoregressive flow
Kingma, D. P., Salimans, T., Jozefowicz, R., Chen, X., Sutskever, I., and Welling, M · 2016
Earlier work this paper cites.
WaveNet: A generative model for raw audio
Original
Oord, A. v. d., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., and Kavukcuoglu, K · 2016
Earlier work this paper cites.

Similar

Neural autoregressive distribution estimation
Uria, B., Côté, M.-A., Gregor, K., Murray, I., and Larochelle, H · 2016
Cited alongside, same era.
Conditional image generation with pixelcnn decoders
Van den Oord, A., Kalchbrenner, N., Espeholt, L., Vinyals, O., Graves, A., et al · 2016
Cited alongside, same era.
Deep Voice: Real-time neural text-to-speech
Arık, S. Ö., Chrzanowski, M., Coates, A., Diamos, G., Gibiansky, A., Kang, Y., Li, X., Miller, J., Ng, A., Raiman, J., Sengupta, S., and Shoeybi, M · 2017
Cited alongside, same era.
A neural parametric singing synthesizer
Blaauw, M. and Bonada, J · 2017
Cited alongside, same era.
Density estimation using Real NVP
Dinh, L., Sohl-Dickstein, J., and Bengio, S · 2017
Cited alongside, same era.
Neural audio synthesis of musical notes with wavenet autoencoders
Engel, J., Resnick, C., Roberts, A., Dieleman, S., Norouzi, M., Eck, D., and Simonyan, K · 2017
Cited alongside, same era.
Deep Voice 2: Multi-speaker neural text-to-speech
Gibiansky, A., Arık, S. Ö., Diamos, G., Miller, J., Peng, K., Ping, W., Raiman, J., and Zhou, Y · 2017
Cited alongside, same era.
Deepbach: a steerable model for bach chorales generation
Hadjeres, G., Pachet, F., and Nielsen, F · 2017
Cited alongside, same era.
Counterpoint by convolution
Huang, C. A., Cooijmans, T., Roberts, A., Courville, A. C., and Eck, D · 2017
Cited alongside, same era.
SampleRNN: An unconditional end-to-end neural audio generation model
Mehri, S., Kumar, K., Gulrajani, I., Kumar, R., Jain, S., Sotelo, J., Courville, A., and Bengio, Y · 2017
Cited alongside, same era.
Neural discrete representation learning
Oord, A. v. d., Vinyals, O., and Kavukcuoglu, K · 2017
Cited alongside, same era.
Masked autoregressive flow for density estimation
Papamakarios, G., Pavlakou, T., and Murray, I · 2017
Cited alongside, same era.

Then

A hierarchical latent vector model for learning long-term structure in music
Roberts, A., Engel, J., Raffel, C., Hawthorne, C., and Eck, D · 2018
Later among the works it cites.
Natural TTS synthesis by conditioning wavenet on mel spectrogram predictions
Shen, J., Pang, R., Weiss, R. J., Schuster, M., Jaitly, N., Yang, Z., Chen, Z., Zhang, Y., Wang, Y., Skerrv-Ryan, R., Saurous, R. A., Agiomvrgiannakis, Y., and Wu, Y · 2018
Later among the works it cites.
VoiceLoop: Voice fitting and synthesis via a phonological loop
Taigman, Y., Wolf, L., Polyak, A., and Nachmani, E · 2018
Later among the works it cites.
Style Tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis
Wang, Y., Stanton, D., Zhang, Y., Skerry-Ryan, R., Battenberg, E., Shor, J., Xiao, Y., Ren, F., Jia, Y., and Saurous, R. A · 2018
Later among the works it cites.
Dota 2 with large scale deep reinforcement learning
Original
Berner, C., Brockman, G., Chan, B., Cheung, V., Dębiak, P., Dennison, C., Farhi, D., Fischer, Q., Hashme, S., Hesse, C., et al · 2019
Later among the works it cites.
Large scale GAN training for high fidelity natural image synthesis
Brock, A., Donahue, J., and Simonyan, K · 2019
Later among the works it cites.
Generating long sequences with sparse transformers
Original
Child, R., Gray, S., Radford, A., and Sutskever, I · 2019
Later among the works it cites.
Hierarchical autoregressive image models with auxiliary decoders
Original
De Fauw, J., Dieleman, S., and Simonyan, K · 2019
Later among the works it cites.
LakhNES: Improving multi-instrumental music generation with cross-domain pre-training
Donahue, C., Mao, H. H., Li, Y. E., Cottrell, G. W., and McAuley, J. J · 2019
Later among the works it cites.
GANSynth: Adversarial neural audio synthesis
Engel, J., Agrawal, K. K., Chen, S., Gulrajani, I., Donahue, C., and Roberts, A · 2019
Later among the works it cites.
Enabling factorized piano music modeling and generation with the MAESTRO dataset
Hawthorne, C., Stasyuk, A., Roberts, A., Simon, I., Huang, C.-Z. A., Dieleman, S., Elsen, E., Engel, J., and Eck, D · 2019
Later among the works it cites.
Spleeter: A fast and state-of-the art music source separation tool with pre-trained models
Hennequin, R., Khlif, A., Voituret, F., and Moussallam, M · 2019
Later among the works it cites.
Axial attention in multidimensional transformers
Original
Ho, J., Kalchbrenner, N., Weissenborn, D., and Salimans, T · 2019
Later among the works it cites.
Hierarchical generative modeling for controllable speech synthesis
Hsu, W.-N., Zhang, Y., Weiss, R. J., Zen, H., Wu, Y., Wang, Y., Cao, Y., Jia, Y., Chen, Z., Shen, J., Nguyen, P., and Pang, R · 2019
Later among the works it cites.
Neural music synthesis for flexible timbre control
Kim, J. W., Bittner, R., Kumar, A., and Bello, J. P · 2019
Later among the works it cites.
MelGAN: Generative adversarial networks for conditional waveform synthesis
Kumar, K., Kumar, R., de Boissiere, T., Gestin, L., Teoh, W. Z., Sotelo, J., de Brébisson, A., Bengio, Y., and Courville, A. C · 2019
Later among the works it cites.
Autoencoder-based music translation
Mor, N., Wolf, L., Polyak, A., and Taigman, Y · 2019
Later among the works it cites.
Clarinet: Parallel wave generation in end-to-end text-to-speech
Ping, W., Peng, K., and Chen, J · 2019
Later among the works it cites.
WaveGlow: A flow-based generative network for speech synthesis
Prenger, R., Valle, R., and Catanzaro, B · 2019
Later among the works it cites.
Generating diverse high-fidelity images with vq-vae-2
Razavi, A., van den Oord, A., and Vinyals, O · 2019
Later among the works it cites.
MelNet: A generative model for audio in the frequency domain
Original
Vasquez, S. and Lewis, M · 2019
Later among the works it cites.
A hierarchical recurrent neural network for symbolic melody generation
Wu, J., Hu, C., Wang, Y., Hu, X., and Zhu, J · 2019
Later among the works it cites.
Self-attention generative adversarial networks
Zhang, H., Goodfellow, I., Metaxas, D., and Odena, A · 2019
Later among the works it cites.
Automatic lyrics transcription in polyphonic music: Does background music help?
Gupta, C., Yılmaz, E., and Li, H · 2020
Closest in time.
Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram
Yamamoto, R., Song, E., and Kim, J.-M · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…