2022

Decomposing neural networks as mappings of correlation functions

Fischer, Kirsten, René, Alexandre, Keup, Christian et al.

Understand

Understanding the functional principles of information processing in deep neural networks continues to be a challenge, in particular for networks with trained and thus non-random weights.

To address this issue, we study the mapping between probability distributions implemented by a deep feed-forward network.
We characterize this mapping as an iterated transformation of distributions, where the non-linearity in each layer transfers information between different orders of correlation functions.
This allows us to identify essential statistics in the data, as well as different information representations that can be used by neural networks.

Built on

C. Fang, J. Lee, P. Yang, and T. Zhang, Modeling from features: a mean-field framework for over-parameterized deep neural networks, in Proceedings of Thirty Fourth Conference on Learning Theory , Proceedings of Machine Learning Research, Vol. 134 (PMLR, 2021) pp. 1887–1936
1936
Earlier work this paper cites.
R. Price, A useful theorem for nonlinear devices having gaussian inputs, IRE Trans. Inf. Theory 4
1958
Earlier work this paper cites.
C. W. Gardiner, Handbook of Stochastic Methods for Physics, Chemistry and the Natural Sciences , 2nd ed., Springer Series in Synergetics No. 13 (Springer-Verlag, Berlin, 1985)
1985
Earlier work this paper cites.
H. Kleinert, Gauge fields in condensed matter, Vol. I , SUPERFLOW AND VORTEX LINES Disorder Fields, Phase Transitions (World Scientific, 1989)
1989
Earlier work this paper cites.
G. Cybenko, Approximation by superpositions of a sigmoidal function, Math. Control Signals. Syst. 2
1989
Earlier work this paper cites.
V. Vapnik, Principles of risk minimization for learning theory, in Adv. Neural Inf. Process. Syst. , Vol. 4, edited by J. Moody, S. Hanson, and R. P. Lippmann (Morgan-Kaufmann, 1992) pp. 831–838
1992
Earlier work this paper cites.
L. Molgedey, J. Schuchhardt, and H. Schuster, Suppressing chaos in neural networks by noise, Phys. Rev. Lett. 69
1992
Earlier work this paper cites.
M. Leshno, V. Y. Lin, A. Pinkus, and S. Schocken, Multilayer feedforward networks with a nonpolynomial activation function can approximate any function, Neural Netw. 6
1993
Earlier work this paper cites.
G. Deco and W. Brauer, Higher order statistical decorrelation without information loss, in Proceedings of the 7th International Conference on Neural Information Processing Systems , NIPS’94 (MIT Press, Cambridge, MA, USA, 1994) pp. 247–254
1994
Earlier work this paper cites.
R. M. Neal, Bayesian Learning for Neural Networks (Springer New York, 1996)
1996
Earlier work this paper cites.
J. Zinn-Justin, Quantum field theory and critical phenomena (Clarendon Press, Oxford, 1996)
1996
Earlier work this paper cites.
R. Kohavi and D. H. Wolpert, Bias plus variance decomposition for zero-one loss functions, in Proceedings of the Thirteenth International Conference on Machine Learning , Vol. 96 (1996) pp. 275–283
1996
Earlier work this paper cites.
C. K. Williams, Computation with infinite neural networks, Neural Comput. 10
1998
Earlier work this paper cites.
V. N. Vapnik, Statistical Learning Theory (Wiley, Hoboken, NJ, USA, 1998)
1998
Earlier work this paper cites.
C. K. I. Williams and D. Barber, Bayesian classification with gaussian processes, IEEE Trans. Pattern Anal. Mach. Intel. 20
1998
Earlier work this paper cites.
S. Blinnikov and R. Moessner, Expansions for nearly gaussian distributions, Astron. Astrophys. Suppl. Ser. 130
1998
Earlier work this paper cites.
A. Pinkus, Approximation theory of the mlp model in neural networks, Acta Numer. 8
1999
Earlier work this paper cites.
A. Papoulis and S. U. Pillai, Probability, Random Variables, and Stochastic Processes , 4th ed. (McGraw-Hill, Boston, 2002)
2002
Earlier work this paper cites.
D. J. MacKay, Information theory, inference and learning algorithms (Cambridge university press, 2003)
2003
Earlier work this paper cites.
C. M. Bishop, Pattern Recognition and Machine Learning (Springer-Verlag New York, Inc., Secaucus, NJ, USA, 2006)
2006
Earlier work this paper cites.

Similar

C. Rasmussen and C. Williams, Gaussian Processes for Machine Learning , Adaptive Computation and Machine Learning (MIT Press, Cambridge, MA, USA, 2006) p. 248
2006
Cited alongside, same era.
C. K. Williams and C. E. Rasmussen, Gaussian Processes for Machine Learning , 1st ed. (MIT Press, Cambridge, 2006)
2006
Cited alongside, same era.
A. Krizhevsky, Learning multiple layers of features from tiny images , Master’s thesis , Department of Computer Science, University of Toronto (2009)
2009
Cited alongside, same era.
Y. LeCun, C. Cortes, and C. J. C. Burges, MNIST handwritten digit database, ATT Labs (2010)
2010
Cited alongside, same era.
A. Krizhevsky, I. Sutskever, and G. E. Hinton, Imagenet classification with deep convolutional neural networks, in Adv. Neural Inf. Process. Syst. , Vol. 25, edited by F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger (Curran Associates, Inc., 2012) pp. 1097–1105
2012
Cited alongside, same era.
D. P. Kingma and J. L. Ba, Adam: A method for stochastic gradient descent, in International Conference on Learning Representations (2015)
Original
2015
Cited alongside, same era.
D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, et al. , Mastering the game of go with deep neural networks and tree search, Nature 529
2016
Cited alongside, same era.
B. Poole, S. Lahiri, M. Raghu, J. Sohl-Dickstein, and S. Ganguli, Exponential expressivity in deep neural networks through transient chaos, in Advances in Neural Information Processing Systems 29 (2016)
2016
Cited alongside, same era.
S. Zagoruyko and N. Komodakis, Wide residual networks, in Proceedings of the British Machine Vision Conference (BMVC) , edited by E. R. H. Richard C. Wilson and W. A. P. Smith (BMVA Press, 2016) pp. 87.1–87.12
2016
Cited alongside, same era.
K. He, X. Zhang, S. Ren, and J. Sun, Deep residual learning for image recognition, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)
2016
Cited alongside, same era.
J. Schuecker, S. Goedeke, D. Dahmen, and M. Helias, Functional methods for disordered neural networks, ArXiv 10.48550/arXiv.1605.06758 (2016), 1605.06758 [cond-mat.dis-nn]
2016
Cited alongside, same era.
H. W. Lin, M. Tegmark, and D. Rolnick, Why does deep and cheap learning work so well?, J. Stat. Phys. 168
2017
Cited alongside, same era.

Then

A. M. Saxe, J. L. McClelland, and S. Ganguli, A mathematical theory of semantic development in deep neural networks, Proc. Natl. Acad. Sci. USA 116
2019
Later among the works it cites.
A. Garriga-Alonso, C. E. Rasmussen, and L. Aitchison, Deep convolutional networks as shallow gaussian processes, in International Conference on Learning Representations (2019)
2019
Later among the works it cites.
I. Loshchilov and F. Hutter, Decoupled weight decay regularization, in International Conference on Learning Representations (2019)
2019
Later among the works it cites.
A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, Pytorch: An imperative style, high-performance deep learning library, in Adv. Neural Inf. Process. Syst. , Vol. 32, edited by H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Curran Associates, Inc., 2019) pp. 8024–8035
2019
Later among the works it cites.
Y. Bahri, J. Kadmon, J. Pennington, S. S. Schoenholz, J. Sohl-Dickstein, and S. Ganguli, Statistical mechanics of deep learning, Annu. Rev. Condens. Matter Phys. 11
2020
Later among the works it cites.
M. Helias and D. Dahmen, Statistical Field Theory for Neural Networks (Springer International Publishing, 2020) p. 203
2020
Later among the works it cites.
E. Dyer and G. Gur-Ari, Asymptotics of wide networks from feynman diagrams, in International Conference on Learning Representations (2020)
2020
Later among the works it cites.
S. Yaida, Non-Gaussian processes and neural networks at finite widths, in Proceedings of The First Mathematical and Scientific Machine Learning Conference , Proceedings of Machine Learning Research, Vol. 107, edited by J. Lu and R. Ward (PMLR, Princeton University, Princeton, NJ, USA, 2020) pp. 165–192
2020
Later among the works it cites.
S. Goldt, M. Mézard, F. Krzakala, and L. Zdeborová, Modeling the Influence of Data Structure on Learning in Neural Networks: The Hidden Manifold Model, Phys. Rev. X 10
2020
Later among the works it cites.
M. E. A. Seddik, C. Louart, M. Tamaazousti, and R. Couillet, Random Matrix Theory Proves that Deep Learning Representations of GAN-data Behave as Gaussian Mixtures, in International Conference on Machine Learning (PMLR, 2020) pp. 8573–8582
2020
Later among the works it cites.
O. Cohen, O. Malka, and Z. Ringel, Learning curves for overparametrized deep neural networks: A field theory perspective, Phys. Rev. Res. 3
2021
Later among the works it cites.
G. Naveh, O. Ben David, H. Sompolinsky, and Z. Ringel, Predicting the outputs of finite deep neural networks trained with noisy gradients, Phys. Rev. E 104
2021
Later among the works it cites.
G. Yang and E. J. Hu, Tensor programs iv: Feature learning in infinite-width neural networks, in Proceedings of the 38th International Conference on Machine Learning , Proceedings of Machine Learning Research, Vol. 139, edited by M. Meila and T. Zhang (PMLR, 2021) pp. 11727–11737
2021
Later among the works it cites.
J. Zhou and H. Huang, Weakly correlated synapses promote dimension reduction in deep neural networks, Phys. Rev. E 103
2021
Later among the works it cites.
S. Goldt, B. Loureiro, G. Reeves, F. Krzakala, M. Mezard, and L. Zdeborova, The gaussian equivalence of generative models for learning with shallow neural networks, in Proceedings of the 2nd Mathematical and Scientific Machine Learning Conference , Proceedings of Machine Learning Research, Vol. 145, edited by J. Bruna, J. Hesthaven, and L. Zdeborova (PMLR, 2022) pp. 426–471
2022
Closest in time.
B. Loureiro, C. Gerbelot, H. Cui, S. Goldt, F. Krzakala, M. Mèzard, and L. Zdeborová, Learning curves of generic features maps for realistic datasets with a teacher-student model, J. Stat. Mech. Theory Exp. 2022
2022
Closest in time.
D. A. Roberts, S. Yaida, and B. Hanin, The Principles of Deep Learning Theory (Cambridge University Press, 2022)
2022
Closest in time.
K. Segadlo, B. Epping, A. van Meegen, D. Dahmen, M. Krämer, and M. Helias, Unified field theoretical approach to deep and recurrent neuronal networks, J. Stat. Mech. Theory Exp. 2022
2022
Closest in time.
B. Bordelon and C. Pehlevan, Self-consistent dynamical field theory of kernel evolution in wide neural networks, arXiv preprint arXiv:2205.09653 (2022)
Original
2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…