Language Models are Universal Embedders

Understand

In the large language model (LLM) revolution, embedding is a key component of various systems, such as retrieving knowledge or memories for LLMs or building content moderation filters.

• As such cases span from English to other natural or programming languages, from retrieval to classification and beyond, it is advantageous to build a unified embedding model rather than dedicated ones for each scenario.
• In this context, the pre-trained multilingual decoder-only large language models, e.g., BLOOM, emerge as a viable backbone option.
• To assess their potential, we propose straightforward strategies for constructing embedders and introduce a universal evaluation benchmark.