Generative Verifiers: Reward Modeling as Next-Token Prediction

Understand

Verifiers or reward models are often used to enhance the reasoning performance of large language models (LLMs).

• A common approach is the Best-of-N method, where N candidate solutions generated by the LLM are ranked by a verifier, and the best one is selected.
• While LLM-based verifiers are typically trained as discriminative classifiers to score solutions, they do not utilize the text generation capabilities of pretrained LLMs.
• To overcome this limitation, we instead propose training verifiers using the ubiquitous next-token prediction objective, jointly on verification and solution generation.