Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic

Understand

Model-free deep reinforcement learning (RL) methods have been successful in a wide variety of simulated domains.

• However, a major obstacle facing deep RL in the real world is their high sample complexity.
• Batch policy gradient methods offer stable learning, but at the cost of high variance, which often requires large batches.
• TD-style methods, such as off-policy actor-critic and Q-learning, are more sample-efficient but biased, and often require costly hyperparameter sweeps to stabilize.