Concise Reasoning via Reinforcement Learning

Understand

A major drawback of reasoning models is their excessive token usage, inflating computational cost, resource demand, and latency.

• We show this verbosity stems not from deeper reasoning but from reinforcement learning loss minimization when models produce incorrect answers.
• With unsolvable problems dominating training, this effect compounds into a systematic tendency toward longer outputs.
• Through theoretical analysis of PPO and GRPO, we prove that incorrect answers inherently drive policies toward verbosity \textit{even when} $\gamma=1$, reframing response lengthening as an optimization artifact.