Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Belief

Understand

Model-based offline reinforcement learning (RL) aims to find highly rewarding policy, by leveraging a previously collected static dataset and a dynamics model.

• While the dynamics model learned through reuse of the static dataset, its generalization ability hopefully promotes policy learning if properly utilized.
• To that end, several works propose to quantify the uncertainty of predicted dynamics, and explicitly apply it to penalize reward.
• However, as the dynamics and the reward are intrinsically different factors in context of MDP, characterizing the impact of dynamics uncertainty through reward penalty may incur unexpected tradeoff between model utilization and risk avoidance.