Value-Based Deep RL Scales Predictably

Understand

Scaling data and compute is critical to the success of modern ML.

• However, scaling demands predictability: we want methods to not only perform well with more compute or data, but also have their performance be predictable from small-scale runs, without running the large-scale experiment.
• In this paper, we show that value-based off-policy RL methods are predictable despite community lore regarding their pathological behavior.
• First, we show that data and compute requirements to attain a given performance level lie on a Pareto frontier, controlled by the updates-to-data (UTD) ratio.