Open Implementation and Study of BEST-RQ for Speech Processing

Understand

Self-Supervised Learning (SSL) has proven to be useful in various speech tasks.

• However, these methods are generally very demanding in terms of data, memory, and computational resources.
• BERT-based Speech pre-Training with Random-projection Quantizer (BEST-RQ), is an SSL method that has shown great performance on Automatic Speech Recognition (ASR) while being simpler than other SSL methods, such as wav2vec 2.0.
• Despite BEST-RQ's great performance, details are lacking in the original paper, such as the amount of GPU/TPU hours used in pre-training, and there is no official easy-to-use open-source implementation.