Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasets

Understand

Offline policy learning is aimed at learning decision-making policies using existing datasets of trajectories without collecting additional data.

• The primary motivation for using reinforcement learning (RL) instead of supervised learning techniques such as behavior cloning is to find a policy that achieves a higher average return than the trajectories constituting the dataset.
• However, we empirically find that when a dataset is dominated by suboptimal trajectories, state-of-the-art offline RL algorithms do not substantially improve over the average return of trajectories in the dataset.
• We argue this is due to an assumption made by current offline RL algorithms of staying close to the trajectories in the dataset.