A Mean Field Theory of Batch Normalization

Understand

We develop a mean field theory for batch normalization in fully-connected feedforward neural networks.

• In so doing, we provide a precise characterization of signal propagation and gradient backpropagation in wide batch-normalized networks at initialization.
• Our theory shows that gradient signals grow exponentially in depth and that these exploding gradients cannot be eliminated by tuning the initial weight variances or by adjusting the nonlinear activation function.
• Indeed, batch normalization itself is the cause of gradient explosion.