Self-Generated Critiques Boost Reward Modeling for Language Models

Understand

Reward modeling is crucial for aligning large language models (LLMs) with human preferences, especially in reinforcement learning from human feedback (RLHF).

• However, current reward models mainly produce scalar scores and struggle to incorporate critiques in a natural language format.
• We hypothesize that predicting both critiques and the scalar reward would improve reward modeling ability.
• Motivated by this, we propose Critic-RM, a framework that improves reward models using self-generated critiques without extra supervision.