T-REG: Preference Optimization with Token-Level Reward Regularization

Understand

Reinforcement learning from human feedback (RLHF) has been crucial in aligning large language models (LLMs) with human values.

• Traditionally, RLHF involves generating responses to a query and using a reward model to assign a reward to the entire response.
• However, this approach faces challenges due to its reliance on a single, sparse reward, which makes it challenging for the model to identify which parts of the sequence contribute most significantly to the final reward.
• Recent methods have attempted to address this limitation by introducing token-level rewards.