Regularized Best-of-N Sampling with Minimum Bayes Risk Objective for Language Model Alignment

Understand

Best-of-N (BoN) sampling with a reward model has been shown to be an effective strategy for aligning Large Language Models (LLMs) to human preferences at the time of decoding.

• BoN sampling is susceptible to a problem known as reward hacking when the accuracy of the reward model is not high enough due to the quality or the quantity of the preference dataset.
• Because the reward model is an imperfect proxy for the true objective, over-optimizing its value can compromise its performance on the true objective.
• In this research, we propose MBR-BoN, a variant of BoN that aims to mitigate reward hacking at inference time by incorporating the Minimum Bayes Risk (MBR) objective as a proximity regularization term.