ETHIC: Evaluating Large Language Models on Long-Context Tasks with High Information Coverage

Understand

Recent advancements in large language models (LLM) capable of processing extremely long texts highlight the need for a dedicated evaluation benchmark to assess their long-context capabilities.

• However, existing methods, like the needle-in-a-haystack test, do not effectively assess whether these models fully utilize contextual information, raising concerns about the reliability of current evaluation techniques.
• To thoroughly examine the effectiveness of existing benchmarks, we introduce a new metric called information coverage (IC), which quantifies the proportion of the input context necessary for answering queries.
• Our findings indicate that current benchmarks exhibit low IC; although the input context may be extensive, the actual usable context is often limited.