Humanity's Last Exam

Understand

Benchmarks are important tools for tracking the rapid advancements in large language model (LLM) capabilities.

• However, benchmarks are not keeping pace in difficulty: LLMs now achieve over 90\% accuracy on popular benchmarks like MMLU, limiting informed measurement of state-of-the-art LLM capabilities.
• In response, we introduce Humanity's Last Exam (HLE), a multi-modal benchmark at the frontier of human knowledge, designed to be the final closed-ended academic benchmark of its kind with broad subject coverage.
• HLE consists of 2,500 questions across dozens of subjects, including mathematics, humanities, and the natural sciences.