ONEBench to Test Them All: Sample-Level Benchmarking Over Open-Ended Capabilities

Understand

Traditional fixed test sets fall short in evaluating open-ended capabilities of foundation models.

• To address this, we propose ONEBench(OpeN-Ended Benchmarking), a new testing paradigm that consolidates individual evaluation datasets into a unified, ever-expanding sample pool.
• ONEBench allows users to generate custom, open-ended evaluation benchmarks from this pool, corresponding to specific capabilities of interest.
• By aggregating samples across test sets, ONEBench enables the assessment of diverse capabilities beyond those covered by the original test sets, while mitigating overfitting and dataset bias.