Variance Reduction for Deep Q-Learning using Stochastic Recursive Gradient

Understand

Deep Q-learning algorithms often suffer from poor gradient estimations with an excessive variance, resulting in unstable training and poor sampling efficiency.

• Stochastic variance-reduced gradient methods such as SVRG have been applied to reduce the estimation variance (Zhao et al.
• 2019).
• However, due to the online instance generation nature of reinforcement learning, directly applying SVRG to deep Q-learning is facing the problem of the inaccurate estimation of the anchor points, which dramatically limits the potentials of SVRG.