Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximation

Understand

We consider the offline reinforcement learning problem, where the aim is to learn a decision making policy from logged data.

• Offline RL -- particularly when coupled with (value) function approximation to allow for generalization in large or continuous state spaces -- is becoming increasingly relevant in practice, because it avoids costly and time-consuming online data collection and is well suited to safety-critical domains.
• Existing sample complexity guarantees for offline value function approximation methods typically require both (1) distributional assumptions (i.e., good coverage) and (2) representational assumptions (i.e., ability to represent some or all $Q$-value functions) stronger than what is required for supervised learning.
• However, the necessity of these conditions and the fundamental limits of offline RL are not well understood in spite of decades of research.