Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networks

Understand

We analyze the dynamics of finite width effects in wide but finite feature learning neural networks.

• Starting from a dynamical mean field theory description of infinite width deep neural network kernel and prediction dynamics, we provide a characterization of the $O(1/\sqrt{\text{width}})$ fluctuations of the DMFT order parameters over random initializations of the network weights.
• Our results, while perturbative in width, unlike prior analyses, are non-perturbative in the strength of feature learning.
• In the lazy limit of network training, all kernels are random but static in time and the prediction variance has a universal form.