Mitigating Value Hallucination in Dyna Planning via Multistep Predecessor Models

Understand

Dyna-style reinforcement learning (RL) agents improve sample efficiency over model-free RL agents by updating the value function with simulated experience generated by an environment model.

• However, it is often difficult to learn accurate models of environment dynamics, and even small errors may result in failure of Dyna agents.
• In this paper, we highlight that one potential cause of that failure is bootstrapping off of the values of simulated states, and introduce a new Dyna algorithm to avoid this failure.
• We discuss a design space of Dyna algorithms, based on using successor or predecessor models -- simulating forwards or backwards -- and using one-step or multi-step updates.