Pretrain Value, Not Reward: Decoupled Value Policy Optimization

Understand

In this paper, we explore how directly pretraining a value model simplifies and stabilizes reinforcement learning from human feedback (RLHF).

• In reinforcement learning, value estimation is the key to policy optimization, distinct from reward supervision.
• The value function predicts the \emph{return-to-go} of a partial answer, that is, how promising the partial answer is if it were continued to completion.
• In RLHF, however, the standard pipeline first pretrains a reward model and then learns a value function online, even though no new reward signals are available once preference data is collected.