Information-Theoretic Considerations in Batch Reinforcement Learning

Understand

Value-function approximation methods that operate in batch mode have foundational importance to reinforcement learning (RL).

• Finite sample guarantees for these methods often crucially rely on two types of assumptions: (1) mild distribution shift, and (2) representation conditions that are stronger than realizability.
• However, the necessity ("why do we need them?") and the naturalness ("when do they hold?") of such assumptions have largely eluded the literature.
• In this paper, we revisit these assumptions and provide theoretical results towards answering the above questions, and make steps towards a deeper understanding of value-function approximation.