Llama2Vec: Unsupervised Adaptation of Large Language Models for Dense Retrieval

Understand

Dense retrieval calls for discriminative embeddings to represent the semantic relationship between query and document.

• It may benefit from the using of large language models (LLMs), given LLMs' strong capability on semantic understanding.
• However, the LLMs are learned by auto-regression, whose working mechanism is completely different from representing whole text as one discriminative embedding.
• Thus, it is imperative to study how to adapt LLMs properly so that they can be effectively initialized as the backbone encoder for dense retrieval.