REBEL: Reward Regularization-Based Approach for Robotic Reinforcement Learning from Human Feedback

Understand

The effectiveness of reinforcement learning (RL) agents in continuous control robotics tasks is mainly dependent on the design of the underlying reward function, which is highly prone to reward hacking.

• A misalignment between the reward function and underlying human preferences (values, social norms) can lead to catastrophic outcomes in the real world especially in the context of robotics for critical decision making.
• Recent methods aim to mitigate misalignment by learning reward functions from human preferences and subsequently performing policy optimization.
• However, these methods inadvertently introduce a distribution shift during reward learning due to ignoring the dependence of agent-generated trajectories on the reward learning objective, ultimately resulting in sub-optimal alignment.