Adaptive Trust Region Policy Optimization: Global Convergence and Faster Rates for Regularized MDPs

Understand

Trust region policy optimization (TRPO) is a popular and empirically successful policy search algorithm in Reinforcement Learning (RL) in which a surrogate problem, that restricts consecutive policies to be 'close' to one another, is iteratively solved.

• Nevertheless, TRPO has been considered a heuristic algorithm inspired by Conservative Policy Iteration (CPI).
• We show that the adaptive scaling mechanism used in TRPO is in fact the natural "RL version" of traditional trust-region methods from convex analysis.
• We first analyze TRPO in the planning setting, in which we have access to the model and the entire state space.