Is Pessimism Provably Efficient for Offline RL?

Understand

We study offline reinforcement learning (RL), which aims to learn an optimal policy based on a dataset collected a priori.

• Due to the lack of further interactions with the environment, offline RL suffers from the insufficient coverage of the dataset, which eludes most existing theoretical analysis.
• In this paper, we propose a pessimistic variant of the value iteration algorithm (PEVI), which incorporates an uncertainty quantifier as the penalty function.
• Such a penalty function simply flips the sign of the bonus function for promoting exploration in online RL, which makes it easily implementable and compatible with general function approximators.