Inverse Reward Design

Understand

Autonomous agents optimize the reward function we give them.

• What they don't know is how hard it is for us to design a reward function that actually captures what we want.
• When designing the reward, we might think of some specific training scenarios, and make sure that the reward will lead to the right behavior in those scenarios.
• Inevitably, agents encounter new scenarios (e.g., new types of terrain) where optimizing that same reward may lead to undesired behavior.