On Random Matrices Arising in Deep Neural Networks: General I.I.D. Case

Understand

We study the distribution of singular values of product of random matrices pertinent to the analysis of deep neural networks.

• The matrices resemble the product of the sample covariance matrices, however, an important difference is that the population covariance matrices assumed to be non-random or random but independent of the random data matrix in statistics and random matrix theory are now certain functions of random data matrices (synaptic weight matrices in the deep neural network terminology).
• The problem has been treated in recent work [25, 13] by using the techniques of free probability theory.
• Since, however, free probability theory deals with population covariance matrices which are independent of the data matrices, its applicability has to be justified.