Optimizing Anytime Reasoning via Budget Relative Policy Optimization

Understand

Scaling test-time compute is crucial for enhancing the reasoning capabilities of large language models (LLMs).

• Existing approaches typically employ reinforcement learning (RL) to maximize a verifiable reward obtained at the end of reasoning traces.
• However, such methods optimize only the final performance under a large and fixed token budget, which hinders efficiency in both training and deployment.
• In this work, we present a novel framework, AnytimeReasoner, to optimize anytime reasoning performance, which aims to improve token efficiency and the flexibility of reasoning under varying token budget constraints.