Sparse Q-learning with Mirror Descent

Understand

This paper explores a new framework for reinforcement learning based on online convex optimization, in particular mirror descent and related algorithms.

• Mirror descent can be viewed as an enhanced gradient method, particularly suited to minimization of convex functions in highdimensional spaces.
• Unlike traditional gradient methods, mirror descent undertakes gradient updates of weights in both the dual space and primal space, which are linked together using a Legendre transform.
• Mirror descent can be viewed as a proximal algorithm where the distance generating function used is a Bregman divergence.