Future-Dependent Value-Based Off-Policy Evaluation in POMDPs

Understand

We study off-policy evaluation (OPE) for partially observable MDPs (POMDPs) with general function approximation.

• Existing methods such as sequential importance sampling estimators and fitted-Q evaluation suffer from the curse of horizon in POMDPs.
• To circumvent this problem, we develop a novel model-free OPE method by introducing future-dependent value functions that take future proxies as inputs.
• Future-dependent value functions play similar roles as classical value functions in fully-observable MDPs.