Composite Q-learning: Multi-scale Q-function Decomposition and Separable Optimization

Understand

In the past few years, off-policy reinforcement learning methods have shown promising results in their application for robot control.

• Deep Q-learning, however, still suffers from poor data-efficiency and is susceptible to stochasticity in the environment or reward functions which is limiting with regard to real-world applications.
• We alleviate these problems by proposing two novel off-policy Temporal-Difference formulations: (1) Truncated Q-functions which represent the return for the first n steps of a target-policy rollout w.r.t.
• the full action-value and (2) Shifted Q-functions, acting as the farsighted return after this truncated rollout.