Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learning

Understand

Reinforcement Learning with Human Feedback (RLHF) has achieved great success in aligning large language models (LLMs) with human preferences.

• Prevalent RLHF approaches are reward-based, following the Bradley-Terry (BT) model assumption, which may not fully capture the complexity of human preferences.
• In this paper, we explore RLHF under a general preference framework and approach it from a game-theoretic perspective.
• Specifically, we formulate the problem as a two-player game and propose a novel online algorithm, iterative Nash policy optimization (INPO).