Dynamic Update-to-Data Ratio: Minimizing World Model Overfitting

Understand

Early stopping based on the validation set performance is a popular approach to find the right balance between under- and overfitting in the context of supervised learning.

• However, in reinforcement learning, even for supervised sub-problems such as world model learning, early stopping is not applicable as the dataset is continually evolving.
• As a solution, we propose a new general method that dynamically adjusts the update to data (UTD) ratio during training based on under- and overfitting detection on a small subset of the continuously collected experience not used for training.
• We apply our method to DreamerV2, a state-of-the-art model-based reinforcement learning algorithm, and evaluate it on the DeepMind Control Suite and the Atari $100$k benchmark.