Data Augmentation using Pre-trained Transformer Models

Understand

Language model based pre-trained models such as BERT have provided significant gains across different NLP tasks.

• In this paper, we study different types of transformer based pre-trained models such as auto-regressive models (GPT-2), auto-encoder models (BERT), and seq2seq models (BART) for conditional data augmentation.
• We show that prepending the class labels to text sequences provides a simple yet effective way to condition the pre-trained models for data augmentation.
• Additionally, on three classification benchmarks, pre-trained Seq2Seq model outperforms other data augmentation methods in a low-resource setting.