Self-training Improves Pre-training for Natural Language Understanding

Understand

Unsupervised pre-training has led to much recent progress in natural language understanding.

• In this paper, we study self-training as another way to leverage unlabeled data through semi-supervised learning.
• To obtain additional data for a specific task, we introduce SentAugment, a data augmentation method which computes task-specific query embeddings from labeled data to retrieve sentences from a bank of billions of unlabeled sentences crawled from the web.
• Unlike previous semi-supervised methods, our approach does not require in-domain unlabeled data and is therefore more generally applicable.