GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignment

Understand

Large Language Models (LLMs) exhibit impressive capabilities but require careful alignment with human preferences.

• Traditional training-time methods finetune LLMs using human preference datasets but incur significant training costs and require repeated training to handle diverse user preferences.
• Test-time alignment methods address this by using reward models (RMs) to guide frozen LLMs without retraining.
• However, existing test-time approaches rely on trajectory-level RMs which are designed to evaluate complete responses, making them unsuitable for autoregressive text generation that requires computing next-token rewards from partial responses.