On the Sample Complexity and Metastability of Heavy-tailed Policy Search in Continuous Control

Understand

Reinforcement learning is a framework for interactive decision-making with incentives sequentially revealed across time without a system dynamics model.

• Due to its scaling to continuous spaces, we focus on policy search where one iteratively improves a parameterized policy with stochastic policy gradient (PG) updates.
• In tabular Markov Decision Problems (MDPs), under persistent exploration and suitable parameterization, global optimality may be obtained.
• By contrast, in continuous space, the non-convexity poses a pathological challenge as evidenced by existing convergence results being mostly limited to stationarity or arbitrary local extrema.