Axioms for AI Alignment from Human Feedback

Understand

In the context of reinforcement learning from human feedback (RLHF), the reward function is generally derived from maximum likelihood estimation of a random utility model based on pairwise comparisons made by humans.

• The problem of learning a reward function is one of preference aggregation that, we argue, largely falls within the scope of social choice theory.
• From this perspective, we can evaluate different aggregation methods via established axioms, examining whether these methods meet or fail well-known standards.
• We demonstrate that both the Bradley-Terry-Luce Model and its broad generalizations fail to meet basic axioms.