Beyond Random Matrix Theory for Deep Networks

Understand

We investigate whether the Wigner semi-circle and Marcenko-Pastur distributions, often used for deep neural network theoretical analysis, match empirically observed spectral densities.

• We find that even allowing for outliers, the observed spectral shapes strongly deviate from such theoretical predictions.
• This raises major questions about the usefulness of these models in deep learning.
• We further show that theoretical results, such as the layered nature of critical points, are strongly dependent on the use of the exact form of these limiting spectral densities.