Asymptotics of Reinforcement Learning with Neural Networks

Understand

We prove that a single-layer neural network trained with the Q-learning algorithm converges in distribution to a random ordinary differential equation as the size of the model and the number of training steps become large.

• Analysis of the limit differential equation shows that it has a unique stationary solution which is the solution of the Bellman equation, thus giving the optimal control for the problem.
• In addition, we study the convergence of the limit differential equation to the stationary solution.
• As a by-product of our analysis, we obtain the limiting behavior of single-layer neural networks when trained on i.i.d.