Autoencoders for music sound modeling: a comparison of linear, shallow, deep, recurrent and variational models

Understand

This study investigates the use of non-linear unsupervised dimensionality reduction techniques to compress a music dataset into a low-dimensional representation which can be used in turn for the synthesis of new sounds.

• We systematically compare (shallow) autoencoders (AEs), deep autoencoders (DAEs), recurrent autoencoders (with Long Short-Term Memory cells -- LSTM-AEs) and variational autoencoders (VAEs) with principal component analysis (PCA) for representing the high-resolution short-term magnitude spectrum of a large and dense dataset of music notes into a lower-dimensional vector (and then convert it back to a magnitude spectrum used for sound resynthesis).
• Our experiments were conducted on the publicly available multi-instrument and multi-pitch database NSynth.
• Interestingly and contrary to the recent literature on image processing, we can show that PCA systematically outperforms shallow AE.