Principled Reinforcement Learning with Human Feedback from Pairwise or $K$-wise Comparisons

Understand

We provide a theoretical framework for Reinforcement Learning with Human Feedback (RLHF).

• Our analysis shows that when the true reward function is linear, the widely used maximum likelihood estimator (MLE) converges under both the Bradley-Terry-Luce (BTL) model and the Plackett-Luce (PL) model.
• However, we show that when training a policy based on the learned reward model, MLE fails while a pessimistic MLE provides policies with improved performance under certain coverage assumptions.
• Additionally, we demonstrate that under the PL model, the true MLE and an alternative MLE that splits the $K$-wise comparison into pairwise comparisons both converge.