Single-Timescale Actor-Critic Provably Finds Globally Optimal Policy

Understand

We study the global convergence and global optimality of actor-critic, one of the most popular families of reinforcement learning algorithms.

• While most existing works on actor-critic employ bi-level or two-timescale updates, we focus on the more practical single-timescale setting, where the actor and critic are updated simultaneously.
• Specifically, in each iteration, the critic update is obtained by applying the Bellman evaluation operator only once while the actor is updated in the policy gradient direction computed using the critic.
• Moreover, we consider two function approximation settings where both the actor and critic are represented by linear or deep neural networks.