REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization

Understand

Reinforcement Learning from Human Feedback~(RLHF) plays a crucial role in aligning Large Language Models~(LLMs).

• The dominant algorithm, Proximal Policy Optimization~(PPO), employs a critic network to estimate advantages, which introduces significant computational and memory overhead.
• To address this, a family of critic-free algorithms (e.g., GRPO, RLOO) has emerged.
• However, these methods typically rely on \textit{prompt-level (local)} advantage normalization, which suffers from inaccurate advantage estimation, a tendency to overfit, and, as we show, is a theoretically biased estimator.