Bayesian Reward Models for LLM Alignment

Understand

To ensure that large language model (LLM) responses are helpful and non-toxic, a reward model trained on human preference data is usually used.

• LLM responses with high rewards are then selected through best-of-$n$ (BoN) sampling or the LLM is further optimized to produce responses with high rewards through reinforcement learning from human feedback (RLHF).
• However, these processes are susceptible to reward overoptimization or `hacking', where responses receive high rewards due to imperfections in the reward model rather than true preference, particularly as prompts or responses deviate from the training data.
• To address these challenges, we propose to train a Bayesian reward model, which signals higher uncertainty further from the training data distribution.