When Your AIs Deceive You: Challenges of Partial Observability in Reinforcement Learning from Human Feedback

Understand

Past analyses of reinforcement learning from human feedback (RLHF) assume that the human evaluators fully observe the environment.

• What happens when human feedback is based only on partial observations? We formally define two failure cases: deceptive inflation and overjustification.
• Modeling the human as Boltzmann-rational w.r.t.
• a belief over trajectories, we prove conditions under which RLHF is guaranteed to result in policies that deceptively inflate their performance, overjustify their behavior to make an impression, or both.