Optimal Design for Reward Modeling in RLHF

Understand

Reinforcement Learning from Human Feedback (RLHF) has become a popular approach to align language models (LMs) with human preferences.

• This method involves collecting a large dataset of human pairwise preferences across various text generations and using it to infer (implicitly or explicitly) a reward model.
• Numerous methods have been proposed to learn the reward model and align a LM with it.
• However, the costly process of collecting human preferences has received little attention and could benefit from theoretical insights.