Understanding Impact of Human Feedback via Influence Functions

Understand

In Reinforcement Learning from Human Feedback (RLHF), it is crucial to learn suitable reward models from human feedback to align large language models (LLMs) with human intentions.

• However, human feedback can often be noisy, inconsistent, or biased, especially when evaluating complex responses.
• Such feedback can lead to misaligned reward signals, potentially causing unintended side effects during the RLHF process.
• To address these challenges, we explore the use of influence functions to measure the impact of human feedback on the performance of reward models.