SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrieval

Understand

In this paper, we propose SimLM (Similarity matching with Language Model pre-training), a simple yet effective pre-training method for dense passage retrieval.

• It employs a simple bottleneck architecture that learns to compress the passage information into a dense vector through self-supervised pre-training.
• We use a replaced language modeling objective, which is inspired by ELECTRA, to improve the sample efficiency and reduce the mismatch of the input distribution between pre-training and fine-tuning.
• SimLM only requires access to unlabeled corpus, and is more broadly applicable when there are no labeled data or queries.