No Preference Left Behind: Group Distributional Preference Optimization

Understand

Preferences within a group of people are not uniform but follow a distribution.

• While existing alignment methods like Direct Preference Optimization (DPO) attempt to steer models to reflect human preferences, they struggle to capture the distributional pluralistic preferences within a group.
• These methods often skew toward dominant preferences, overlooking the diversity of opinions, especially when conflicting preferences arise.
• To address this issue, we propose Group Distributional Preference Optimization (GDPO), a novel framework that aligns language models with the distribution of preferences within a group by incorporating the concept of beliefs that shape individual preferences.