What Makes a Reward Model a Good Teacher? An Optimization Perspective

Understand

The success of Reinforcement Learning from Human Feedback (RLHF) critically depends on the quality of the reward model.

• However, while this quality is primarily evaluated through accuracy, it remains unclear whether accuracy fully captures what makes a reward model an effective teacher.
• We address this question from an optimization perspective.
• First, we prove that regardless of how accurate a reward model is, if it induces low reward variance, then the RLHF objective suffers from a flat landscape.