EnCodecMAE: Leveraging neural codecs for universal audio representation learning

Understand

The goal of universal audio representation learning is to obtain foundational models that can be used for a variety of downstream tasks involving speech, music and environmental sounds.

• To approach this problem, methods inspired by works on self-supervised learning for NLP, like BERT, or computer vision, like masked autoencoders (MAE), are often adapted to the audio domain.
• In this work, we propose masking representations of the audio signal, and training a MAE to reconstruct the masked segments.
• The reconstruction is done by predicting the discrete units generated by EnCodec, a neural audio codec, from the unmasked inputs.