Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval

Understand

Conducting text retrieval in a dense learned representation space has many intriguing advantages over sparse retrieval.

• Yet the effectiveness of dense retrieval (DR) often requires combination with sparse retrieval.
• In this paper, we identify that the main bottleneck is in the training mechanisms, where the negative instances used in training are not representative of the irrelevant documents in testing.
• This paper presents Approximate nearest neighbor Negative Contrastive Estimation (ANCE), a training mechanism that constructs negatives from an Approximate Nearest Neighbor (ANN) index of the corpus, which is parallelly updated with the learning process to select more realistic negative training instances.