Pre-training Text Representations as Meta Learning

Understand

Pre-training text representations has recently been shown to significantly improve the state-of-the-art in many natural language processing tasks.

• The central goal of pre-training is to learn text representations that are useful for subsequent tasks.
• However, existing approaches are optimized by minimizing a proxy objective, such as the negative log likelihood of language modeling.
• In this work, we introduce a learning algorithm which directly optimizes model's ability to learn text representations for effective learning of downstream tasks.