Seizing Serendipity: Exploiting the Value of Past Success in Off-Policy Actor-Critic

Understand

Learning high-quality $Q$-value functions plays a key role in the success of many modern off-policy deep reinforcement learning (RL) algorithms.

• Previous works primarily focus on addressing the value overestimation issue, an outcome of adopting function approximators and off-policy learning.
• Deviating from the common viewpoint, we observe that $Q$-values are often underestimated in the latter stage of the RL training process, potentially hindering policy learning and reducing sample efficiency.
• We find that such a long-neglected phenomenon is often related to the use of inferior actions from the current policy in Bellman updates as compared to the more optimal action samples in the replay buffer.