Self-supervised speech representation learning for keyword-spotting with light-weight transformers

Understand

Self-supervised speech representation learning (S3RL) is revolutionizing the way we leverage the ever-growing availability of data.

• While S3RL related studies typically use large models, we employ light-weight networks to comply with tight memory of compute-constrained devices.
• We demonstrate the effectiveness of S3RL on a keyword-spotting (KS) problem by using transformers with 330k parameters and propose a mechanism to enhance utterance-wise distinction, which proves crucial for improving performance on classification tasks.
• On the Google speech commands v2 dataset, the proposed method applied to the Auto-Regressive Predictive Coding S3RL led to a 1.2% accuracy improvement compared to training from scratch.