Beyond RLHF and NLHF: Population-Proportional Alignment under an Axiomatic Framework

Understand

Conventional preference learning methods often prioritize opinions held more widely when aggregating preferences from multiple evaluators.

• This may result in policies that are biased in favor of some types of opinions or groups and susceptible to strategic manipulation.
• To address this issue, we develop a novel preference learning framework capable of aligning aggregate opinions and policies proportionally with the true population distribution of evaluator preferences.
• Grounded in social choice theory, our approach infers the feasible set of evaluator population distributions directly from pairwise comparison data.