Shorten After You're Right: Lazy Length Penalties for Reasoning RL

Understand

Large reasoning models, such as OpenAI o1 or DeepSeek R1, have demonstrated remarkable performance on reasoning tasks but often incur a long reasoning path with significant memory and time costs.

• Existing methods primarily aim to shorten reasoning paths by introducing additional training data and stages.
• In this paper, we propose three critical reward designs integrated directly into the reinforcement learning process of large reasoning models, which reduce the response length without extra training stages.
• Experiments on four settings show that our method significantly decreases response length while maintaining or even improving performance.