The Curious Price of Distributional Robustness in Reinforcement Learning with a Generative Model

Understand

This paper investigates model robustness in reinforcement learning (RL) to reduce the sim-to-real gap in practice.

• We adopt the framework of distributionally robust Markov decision processes (RMDPs), aimed at learning a policy that optimizes the worst-case performance when the deployed environment falls within a prescribed uncertainty set around the nominal MDP.
• Despite recent efforts, the sample complexity of RMDPs remained mostly unsettled regardless of the uncertainty set in use.
• It was unclear if distributional robustness bears any statistical consequences when benchmarked against standard RL.