CoBERT: Self-Supervised Speech Representation Learning Through Code Representation Learning

Understand

Speech is the surface form of a finite set of phonetic units, which can be represented by discrete codes.

• We propose the Code BERT (CoBERT) approach for self-supervised speech representation learning.
• The idea is to convert an utterance to a sequence of discrete codes, and perform code representation learning, where we predict the code representations based on a masked view of the original speech input.
• Unlike the prior self-distillation approaches of which the teacher and the student are of the same modality, our target model predicts representations from a different modality.