Efficient Evaluation of Natural Stochastic Policies in Offline Reinforcement Learning

Understand

We study the efficient off-policy evaluation of natural stochastic policies, which are defined in terms of deviations from the behavior policy.

• This is a departure from the literature on off-policy evaluation where most work consider the evaluation of explicitly specified policies.
• Crucially, offline reinforcement learning with natural stochastic policies can help alleviate issues of weak overlap, lead to policies that build upon current practice, and improve policies' implementability in practice.
• Compared with the classic case of a pre-specified evaluation policy, when evaluating natural stochastic policies, the efficiency bound, which measures the best-achievable estimation error, is inflated since the evaluation policy itself is unknown.