2018

Generative timbre spaces: regularizing variational auto-encoders with perceptual metrics

Esling, Philippe, Chemla--Romeu-Santos, Axel, Bitton, Adrien

Understand

Timbre spaces have been used in music perception to study the perceptual relationships between instruments based on dissimilarity ratings.

However, these spaces do not generalize to novel examples and do not provide an invertible mapping, preventing audio synthesis.
In parallel, generative models have aimed to provide methods for synthesizing novel timbres.
However, these systems do not provide an understanding of their inner workings and are usually not related to any perceptually relevant information.

Built on

“Monte carlo sampling methods using markov chains and their applications,”
Keith Hastings, · 1970
Earlier work this paper cites.
“Multidimensional perceptual scaling of musical timbres,”
John M Grey, · 1977
Earlier work this paper cites.
“Perceptual effects of spectral modifications on musical timbres,”
John M Grey and John W Gordon, · 1978
Earlier work this paper cites.
“Why is musical timbre so hard to understand,”
Carol L Krumhansl, · 1989
Earlier work this paper cites.
“Isolating the dynamic attributes of musical timbrea,”
Paul Iverson and Carol L Krumhansl, · 1993
Earlier work this paper cites.
“Perceptual scaling of synthesized musical timbres: Common dimensions, specificities, and latent subject classes,”
Stephen McAdams, Suzanne Winsberg, Sophie Donnadieu, Geert De Soete, and Jochen Krimphoff, · 1995
Earlier work this paper cites.
“Studio online 3.0: An internet "killer application" for remote access to ircam sounds and processing tools,”
Guillaume Ballet, Riccardo Borghesi, Peter Hoffmann, and Fabien Levy, · 1999
Earlier work this paper cites.

Similar

“A common perceptual space for harmonic and percussive timbres,”
Stephen Lakatos, · 2000
Cited alongside, same era.
“A meta-analysis of acoustic correlates of timbre dimensions,”
Stephen McAdams, Bruno L. Giordano, Patrick Susini, Geoffroy Peeters, and Vincent Rioux, · 2006
Cited alongside, same era.
“Visualizing data using t-sne,”
Laurens van der Maaten and Geoffrey Hinton, · 2008
Cited alongside, same era.
“Theory, implementation and applications of nonstationary gabor frames,”
Peter Balazs, Monika Dörfler, Florent Jaillet, Nicki Holighaus, and G Velasco, · 2011
Cited alongside, same era.
“Auto-encoding variational bayes,”
Diederik P Kingma and Max Welling, · 2014
Cited alongside, same era.
“Pattern recognition and machine learning,”
Christopher M Bishop and Tom M Mitchell, · 2014
Cited alongside, same era.
“Adam: A method for stochastic optimization,”
Diederik P Kingma and Jimmy Ba, · 2014
Cited alongside, same era.

Then

“Wavenet: A generative model for raw audio,”
Original
Aaron Van Den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu, · 2016
Later among the works it cites.
“beta-vae: Learning basic visual concepts with a constrained variational framework,”
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner, · 2016
Later among the works it cites.
“How to train deep variational autoencoders and probabilistic ladder networks,”
Original
Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, and Ole Winther, · 2016
Later among the works it cites.
“Samplernn: An unconditional end-to-end neural audio generation model,”
Soroush Mehri, Kundan Kumar, Ishaan Gulrajani, Rithesh Kumar, Shubham Jain, Jose Sotelo, Aaron Courville, and Yoshua Bengio, · 2017
Later among the works it cites.
“Neural audio synthesis of musical notes with wavenet autoencoders,”
Jesse Engel, Cinjon Resnick, Adam Roberts, Sander Dieleman, Douglas Eck, Karen Simonyan, and Mohammad Norouzi, · 2017
Later among the works it cites.
“Variational recurrent neural networks for speech separation,”
Jen-Tzung Kuo and Kuan-Ting Chien, · 2017
Later among the works it cites.
“Learning latent representations for speech generation and transformation,”
Original
Wei-Ning Hsu, Yu Zhang, and James Glass, · 2017
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…