Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback

Understand

Aligning the behavior of Large language models (LLMs) with human intentions and values remains a critical challenge.

• Reinforcement learning from human feedback (RLHF) aligns LLMs by training a reward model (RM) on human preferences and fine-tuning the LLMs to maximize RM feedback.
• Despite its effectiveness and popularity, RLHF is prone to biased local optimization.
• It means RM fails to provide feedback that accurately aligns with human preference, causing LLMs to explore unexpected generalizations, and failing to achieve alignment objectives.