Dynamical Isometry is Achieved in Residual Networks in a Universal Way for any Activation Function

Understand

We demonstrate that in residual neural networks (ResNets) dynamical isometry is achievable irrespectively of the activation function used.

• We do that by deriving, with the help of Free Probability and Random Matrix Theories, a universal formula for the spectral density of the input-output Jacobian at initialization, in the large network width and depth limit.
• The resulting singular value spectrum depends on a single parameter, which we calculate for a variety of popular activation functions, by analyzing the signal propagation in the artificial neural network.
• We corroborate our results with numerical simulations of both random matrices and ResNets applied to the CIFAR-10 classification problem.