Learning Pessimism for Robust and Efficient Off-Policy Reinforcement Learning

Understand

Off-policy deep reinforcement learning algorithms commonly compensate for overestimation bias during temporal-difference learning by utilizing pessimistic estimates of the expected target returns.

• In this work, we propose Generalized Pessimism Learning (GPL), a strategy employing a novel learnable penalty to enact such pessimism.
• In particular, we propose to learn this penalty alongside the critic with dual TD-learning, a new procedure to estimate and minimize the magnitude of the target returns bias with trivial computational cost.
• GPL enables us to accurately counteract overestimation bias throughout training without incurring the downsides of overly pessimistic targets.