RepLiQA: A Question-Answering Dataset for Benchmarking LLMs on Unseen Reference Content

Understand

Large Language Models (LLMs) are trained on vast amounts of data, most of which is automatically scraped from the internet.

• This data includes encyclopedic documents that harbor a vast amount of general knowledge (e.g., Wikipedia) but also potentially overlap with benchmark datasets used for evaluating LLMs.
• Consequently, evaluating models on test splits that might have leaked into the training set is prone to misleading conclusions.
• To foster sound evaluation of language models, we introduce a new test dataset named RepLiQA, suited for question-answering and topic retrieval tasks.