MAD-TD: Model-Augmented Data stabilizes High Update Ratio RL

Understand

Building deep reinforcement learning (RL) agents that find a good policy with few samples has proven notoriously challenging.

• To achieve sample efficiency, recent work has explored updating neural networks with large numbers of gradient steps for every new sample.
• While such high update-to-data (UTD) ratios have shown strong empirical performance, they also introduce instability to the training process.
• Previous approaches need to rely on periodic neural network parameter resets to address this instability, but restarting the training process is infeasible in many real-world applications and requires tuning the resetting interval.