Surrogate Signals from Format and Length: Reinforcement Learning for Solving Mathematical Problems without Ground Truth Answers

Understand

Large Language Models (LLMs) have achieved remarkable success in natural language processing tasks, with Reinforcement Learning (RL) playing a key role in adapting them to specific applications.

• In mathematical problem solving, however, the reliance on ground truth answers poses significant challenges due to their high collection cost and limited availability.
• This work explores the use of simple surrogate signals, format and length, to guide RL training.
• We find that early training is dominated by format learning, where structural feedback alone accounts for most performance gains.