Tensor Programs III: Neural Matrix Laws

Understand

In a neural network (NN), *weight matrices* linearly transform inputs into *preactivations* that are then transformed nonlinearly into *activations*.

• A typical NN interleaves multitudes of such linear and nonlinear transforms to express complex functions.
• Thus, the (pre-)activations depend on the weights in an intricate manner.
• We show that, surprisingly, (pre-)activations of a randomly initialized NN become *independent* from the weights as the NN's widths tend to infinity, in the sense of asymptotic freeness in random matrix theory.