Off-policy Evaluation in Infinite-Horizon Reinforcement Learning with Latent Confounders

Understand

Off-policy evaluation (OPE) in reinforcement learning is an important problem in settings where experimentation is limited, such as education and healthcare.

• But, in these very same settings, observed actions are often confounded by unobserved variables making OPE even more difficult.
• We study an OPE problem in an infinite-horizon, ergodic Markov decision process with unobserved confounders, where states and actions can act as proxies for the unobserved confounders.
• We show how, given only a latent variable model for states and actions, policy value can be identified from off-policy data.