ReLOAD: Reinforcement Learning with Optimistic Ascent-Descent for Last-Iterate Convergence in Constrained MDPs

Understand

In recent years, Reinforcement Learning (RL) has been applied to real-world problems with increasing success.

• Such applications often require to put constraints on the agent's behavior.
• Existing algorithms for constrained RL (CRL) rely on gradient descent-ascent, but this approach comes with a caveat.
• While these algorithms are guaranteed to converge on average, they do not guarantee last-iterate convergence, i.e., the current policy of the agent may never converge to the optimal solution.