Interpolated Policy Gradient: Merging On-Policy and Off-Policy Gradient Estimation for Deep Reinforcement Learning

Understand

Off-policy model-free deep reinforcement learning methods using previously collected data can improve sample efficiency over on-policy policy gradient techniques.

• On the other hand, on-policy algorithms are often more stable and easier to use.
• This paper examines, both theoretically and empirically, approaches to merging on- and off-policy updates for deep reinforcement learning.
• Theoretical results show that off-policy updates with a value function estimator can be interpolated with on-policy policy gradient updates whilst still satisfying performance bounds.