Measuring AI Ability to Complete Long Software Tasks

Understand

Despite rapid progress on AI benchmarks, the real-world meaning of benchmark performance remains unclear.

• To quantify the capabilities of AI systems in terms of human capabilities, we propose a new metric: 50%-task-completion time horizon.
• This is the time humans typically take to complete tasks that AI models can complete with 50% success rate.
• We first timed humans with relevant domain expertise on a combination of RE-Bench, HCAST, and 66 novel shorter tasks.