Semiparametric Off-Policy Inference for Optimal Policy Values under Possible Non-Uniqueness

Understand

Off-policy evaluation (OPE) constructs confidence intervals for the value of a target policy using data generated under a different behavior policy.

• Most existing inference methods focus on fixed target policies and may fail when the target policy is estimated as optimal, particularly when the optimal policy is non-unique or nearly deterministic.
• We study inference for the value of optimal policies in Markov decision processes.
• In an auxiliary augmented transition-sampling experiment, we characterize the existence of the efficient influence function and show that non-regularity arises when competing optimal policies havedistinct first-order gradients.