ATEB: Evaluating and Improving Advanced NLP Tasks for Text Embedding Models

Understand

Traditional text embedding benchmarks primarily evaluate embedding models' capabilities to capture semantic similarity.

• However, more advanced NLP tasks require a deeper understanding of text, such as safety and factuality.
• These tasks demand an ability to comprehend and process complex information, often involving the handling of sensitive content, or the verification of factual statements against reliable sources.
• We introduce a new benchmark designed to assess and highlight the limitations of embedding models trained on existing information retrieval data mixtures on advanced capabilities, which include factuality, safety, instruction following, reasoning and document-level understanding.