The Accuracy Paradox in RLHF: When Better Reward Models Don't Yield Better Language Models

Understand

Reinforcement Learning from Human Feedback significantly enhances Natural Language Processing by aligning language models with human expectations.

• A critical factor in this alignment is the strength of reward models used during training.
• This study explores whether stronger reward models invariably lead to better language models.
• In this paper, through experiments on relevance, factuality, and completeness tasks using the QA-FEEDBACK dataset and reward models based on Longformer, we uncover a surprising paradox: language models trained with moderately accurate reward models outperform those guided by highly accurate ones.