Offline Reinforcement Learning with Imbalanced Datasets

Understand

The prevalent use of benchmarks in current offline reinforcement learning (RL) research has led to a neglect of the imbalance of real-world dataset distributions in the development of models.

• The real-world offline RL dataset is often imbalanced over the state space due to the challenge of exploration or safety considerations.
• In this paper, we specify properties of imbalanced datasets in offline RL, where the state coverage follows a power law distribution characterized by skewed policies.
• Theoretically and empirically, we show that typically offline RL methods based on distributional constraints, such as conservative Q-learning (CQL), are ineffective in extracting policies under the imbalanced dataset.