Offline Reinforcement Learning with Pseudometric Learning

Understand

Offline Reinforcement Learning methods seek to learn a policy from logged transitions of an environment, without any interaction.

• In the presence of function approximation, and under the assumption of limited coverage of the state-action space of the environment, it is necessary to enforce the policy to visit state-action pairs close to the support of logged transitions.
• In this work, we propose an iterative procedure to learn a pseudometric (closely related to bisimulation metrics) from logged transitions, and use it to define this notion of closeness.
• We show its convergence and extend it to the function approximation setting.