Provable Pluralistic Alignment: Multi-Party RLHF under Offline Human Feedback

Understand

Pluralistic alignment requires learning from feedback that reflects persistent and potentially conflicting stakeholder preferences while ultimately selecting a single collective policy.

• We study this problem in offline reinforcement learning from human feedback (RLHF), where the party associated with each comparison is observed.
• Under a shared low-rank linear reward model, we jointly estimate party-specific rewards and perform pessimistic policy optimization under Nash, Utilitarian, and Egalitarian social-welfare objectives.
• We establish nonasymptotic bounds for party-specific reward estimation and the resulting policy suboptimality under offline coverage conditions.