Off-policy Evaluation with Deeply-abstracted States

Understand

Off-policy evaluation (OPE) is crucial for assessing a target policy's impact offline before its deployment.

• However, achieving accurate OPE in large state spaces remains challenging.
• This paper studies state abstractions -- originally designed for policy learning -- in the context of OPE.
• Our contributions are three-fold: (i) We define a set of irrelevance conditions central to learning state abstractions for OPE, and derive a backward-model-irrelevance condition for achieving irrelevance in %sequential and (marginalized) importance sampling ratios by constructing a time-reversed Markov decision process (MDP).