PILAF: Optimal Human Preference Sampling for Reward Modeling

Understand

As large language models increasingly drive real-world applications, aligning them with human values becomes paramount.

• Reinforcement Learning from Human Feedback (RLHF) has emerged as a key technique, translating preference data into reward models when oracle human values remain inaccessible.
• In practice, RLHF mostly relies on approximate reward models, which may not consistently guide the policy toward maximizing the underlying human values.
• We propose Policy-Interpolated Learning for Aligned Feedback (PILAF), a novel response sampling strategy for preference labeling that explicitly aligns preference learning with maximizing the underlying oracle reward.