Pre-train a Discriminative Text Encoder for Dense Retrieval via Contrastive Span Prediction

Understand

Dense retrieval has shown promising results in many information retrieval (IR) related tasks, whose foundation is high-quality text representation learning for effective search.

• Some recent studies have shown that autoencoder-based language models are able to boost the dense retrieval performance using a weak decoder.
• However, we argue that 1) it is not discriminative to decode all the input texts and, 2) even a weak decoder has the bypass effect on the encoder.
• Therefore, in this work, we introduce a novel contrastive span prediction task to pre-train the encoder alone, but still retain the bottleneck ability of the autoencoder.