Reward Shaping to Mitigate Reward Hacking in RLHF

Understand

Reinforcement learning from human feedback (RLHF) is widely used to align large language models (LLMs) with human preferences.

• However, RLHF remains vulnerable to \emph{reward hacking}, whereby a policy exploits imperfections in the reward function instead of learning the intended behavior, thereby undermining alignment.
• Although reward shaping can stabilize RLHF training and partially mitigate reward hacking, shaping methods and their underlying design principles have not been systematically investigated.
• To address this gap, we conduct a comprehensive study of prevalent reward-shaping techniques.