Hallucinated Adversarial Control for Conservative Offline Policy Evaluation

Understand

We study the problem of conservative off-policy evaluation (COPE) where given an offline dataset of environment interactions, collected by other agents, we seek to obtain a (tight) lower bound on a policy's performance.

• This is crucial when deciding whether a given policy satisfies certain minimal performance/safety criteria before it can be deployed in the real world.
• To this end, we introduce HAMBO, which builds on an uncertainty-aware learned model of the transition dynamics.
• To form a conservative estimate of the policy's performance, HAMBO hallucinates worst-case trajectories that the policy may take, within the margin of the models' epistemic confidence regions.