Finite size corrections for neural network Gaussian processes

Understand

There has been a recent surge of interest in modeling neural networks (NNs) as Gaussian processes.

• In the limit of a NN of infinite width the NN becomes equivalent to a Gaussian process.
• Here we demonstrate that for an ensemble of large, finite, fully connected networks with a single hidden layer the distribution of outputs at initialization is well described by a Gaussian perturbed by the fourth Hermite polynomial for weights drawn from a symmetric distribution.
• We show that the scale of the perturbation is inversely proportional to the number of units in the NN and that higher order terms decay more rapidly, thereby recovering the Edgeworth expansion.