A Dynamical Central Limit Theorem for Shallow Neural Networks

Understand

Recent theoretical works have characterized the dynamics of wide shallow neural networks trained via gradient descent in an asymptotic mean-field limit when the width tends towards infinity.

• At initialization, the random sampling of the parameters leads to deviations from the mean-field limit dictated by the classical Central Limit Theorem (CLT).
• However, since gradient descent induces correlations among the parameters, it is of interest to analyze how these fluctuations evolve.
• Here, we use a dynamical CLT to prove that the asymptotic fluctuations around the mean limit remain bounded in mean square throughout training.