A Two-Time-Scale Stochastic Optimization Framework with Applications in Control and Reinforcement Learning

Understand

We study a new two-time-scale stochastic gradient method for solving optimization problems, where the gradients are computed with the aid of an auxiliary variable under samples generated by time-varying MDPs controlled by the underlying optimization variable.

• These time-varying samples make gradient directions in our update biased and dependent, which can potentially lead to the divergence of the iterates.
• In our two-time-scale approach, one scale is to estimate the true gradient from these samples, which is then used to update the estimate of the optimal solution.
• While these two iterates are implemented simultaneously, the former is updated "faster" than the latter.