Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty

Understand

When language models (LMs) are trained via reinforcement learning (RL) to generate natural language "reasoning chains", their performance improves on a variety of difficult question answering tasks.

• Today, almost all successful applications of RL for reasoning use binary reward functions that evaluate the correctness of LM outputs.
• Because such reward functions do not penalize guessing or low-confidence outputs, they often have the unintended side-effect of degrading calibration and increasing the rate at which LMs generate incorrect responses (or "hallucinate") in other problem domains.
• This paper describes RLCR (Reinforcement Learning with Calibration Rewards), an approach to training reasoning models that jointly improves accuracy and calibrated confidence estimation.