The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs

Understand

Does continued scaling of large language models (LLMs) yield diminishing returns? In this work, we show that short-task benchmarks may give an illusion of slowing progress, as even marginal gains in single-step accuracy can compound into exponential improvements in the length of tasks a model can successfully complete.

• Then, we argue that failures of LLMs when simple tasks are made longer arise from mistakes in execution, rather than an inability to reason.
• So, we propose isolating execution capability, by explicitly providing the knowledge and plan needed to solve a long-horizon task.
• First, we find that larger models can correctly execute significantly more turns even when small models have near-perfect single-turn accuracy.