Improving Reward Models with Synthetic Critiques

Understand

Reward models (RMs) play a critical role in aligning language models through the process of reinforcement learning from human feedback.

• RMs are trained to predict a score reflecting human preference, which requires significant time and cost for human annotation.
• Additionally, RMs tend to quickly overfit on superficial features in the training set, hindering their generalization performance on unseen distributions.
• We propose a novel approach using synthetic natural language critiques generated by large language models to provide additional feedback, evaluating aspects such as instruction following, correctness, and style.