General-Purpose Speech Representation Learning through a Self-Supervised Multi-Granularity Framework

Understand

This paper presents a self-supervised learning framework, named MGF, for general-purpose speech representation learning.

• In the design of MGF, speech hierarchy is taken into consideration.
• Specifically, we propose to use generative learning approaches to capture fine-grained information at small time scales and use discriminative learning approaches to distill coarse-grained or semantic information at large time scales.
• For phoneme-scale learning, we borrow idea from the masked language model but tailor it for the continuous speech signal by replacing classification loss with a contrastive loss.