Low-Rank Contextual Reinforcement Learning from Heterogeneous Human Feedback

Understand

Reinforcement learning from human feedback (RLHF) has become a cornerstone for aligning large language models with human preferences.

• However, the heterogeneity of human feedback, driven by diverse individual contexts and preferences, poses significant challenges for reward learning.
• To address this, we propose a Low-rank Contextual RLHF (LoCo-RLHF) framework that integrates contextual information to better model heterogeneous feedback while maintaining computational efficiency.
• Our approach builds on a contextual preference model, leveraging the intrinsic low-rank structure of the interaction between user contexts and query-answer pairs to mitigate the high dimensionality of feature representations.