Adaptive Rollout Length for Model-Based RL Using Model-Free Deep RL

Understand

Model-based reinforcement learning promises to learn an optimal policy from fewer interactions with the environment compared to model-free reinforcement learning by learning an intermediate model of the environment in order to predict future interactions.

• When predicting a sequence of interactions, the rollout length, which limits the prediction horizon, is a critical hyperparameter as accuracy of the predictions diminishes in the regions that are further away from real experience.
• As a result, with a longer rollout length, an overall worse policy is learned in the long run.
• Thus, the hyperparameter provides a trade-off between quality and efficiency.