RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encoder

Understand

Despite pre-training's progress in many important NLP tasks, it remains to explore effective pre-training strategies for dense retrieval.

• In this paper, we propose RetroMAE, a new retrieval oriented pre-training paradigm based on Masked Auto-Encoder (MAE).
• RetroMAE is highlighted by three critical designs.
• 1) A novel MAE workflow, where the input sentence is polluted for encoder and decoder with different masks.