Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism

Understand

Offline (or batch) reinforcement learning (RL) algorithms seek to learn an optimal policy from a fixed dataset without active data collection.

• Based on the composition of the offline dataset, two main categories of methods are used: imitation learning which is suitable for expert datasets and vanilla offline RL which often requires uniform coverage datasets.
• From a practical standpoint, datasets often deviate from these two extremes and the exact data composition is usually unknown a priori.
• To bridge this gap, we present a new offline RL framework that smoothly interpolates between the two extremes of data composition, hence unifying imitation learning and vanilla offline RL.