Proximal Point Nash Learning from Human Feedback

Understand

Traditional Reinforcement Learning from Human Feedback (RLHF) often relies on reward models, frequently assuming preference structures like the Bradley--Terry model, which may not accurately capture the complexities of real human preferences (e.g., intransitivity).

• Nash Learning from Human Feedback (NLHF) offers a more direct alternative by framing the problem as finding a Nash equilibrium of a game defined by these preferences.
• While many works study the Nash learning problem directly in the policy space, we instead consider it under a more realistic policy parametrization setting.
• We first analyze a simple self-play policy gradient method, which is equivalent to Online IPO.