UniSpeech at scale: An Empirical Study of Pre-training Method on Large-Scale Speech Recognition Dataset

Understand

Recently, there has been a vast interest in self-supervised learning (SSL) where the model is pre-trained on large scale unlabeled data and then fine-tuned on a small labeled dataset.

• The common wisdom is that SSL helps resource-limited tasks in which only a limited amount of labeled data is available.
• The benefit of SSL keeps diminishing when the labeled training data amount increases.
• To our best knowledge, at most a few thousand hours of labeled data was used in the study of SSL.