Fetching the paper…
Reading the bibliography…
Given two or more Deep Neural Networks (DNNs) with the same or similar architectures, and trained on the same dataset, but trained with different solvers, parameters, hyper-parameters, regularization, etc., can we predict which DNN will have the best test accuracy, and can we do so without peeking at the test data? In this paper, we show how to use a new Theory of Heavy-Tailed Self-Regularization (HT-SR) to answer this.
Theory of Lévy matrices
P. Cizeau and J. P. Bouchaud · 1994
Earlier work this paper cites.
For valid generalization, the size of the weights is more important than the size of the network
P. L. Bartlett · 1997
Earlier work this paper cites.
Universality classes for extreme-value statistics
J.-P. Bouchaud and M. Mézard · 1997
Earlier work this paper cites.
Statistical mechanics of learning
A. Engel and C. P. L. Van den Broeck · 2001
Earlier work this paper cites.
Statistical Physics of Spin Glasses and Information Processing: An Introduction
H. Nishimori · 2001
Earlier work this paper cites.
Lévy matrices and financial covariances
Z. Burda, J. Jurkiewicz, M. A. Nowak, G. Papp, and I. Zahed · 2001
Earlier work this paper cites.
Theory of Financial Risk and Derivative Pricing: From Statistical Physics to Risk Management
J. P. Bouchaud and M. Potters · 2003
Earlier work this paper cites.
Power laws, Pareto distributions and Zipf’s law
M. E. J. Newman · 2005
Earlier work this paper cites.
Random matrix theory
A. Edelman and N. R. Rao · 2005
Earlier work this paper cites.
Critical phenomena in natural sciences: chaos, fractals, selforganization and disorder: concepts and tools
D. Sornette · 2006
Earlier work this paper cites.
Random Lévy matrices revisited
Z. Burda, J. Jurkiewicz, M. A. Nowak, G. Papp, and I. Zahed · 2006
Earlier work this paper cites.
Hausdorff dimension, its properties, and its surprises
D. Schleicher · 2007
Earlier work this paper cites.
On the top eigenvalue of heavy-tailed random matrices
G. Biroli, J.-P. Bouchaud, and M. Potters · 2007
Earlier work this paper cites.
Extreme value problems in random matrix theory and other disordered systems
G. Biroli, J.-P. Bouchaud, and M. Potters · 2007
Earlier work this paper cites.
Heavy-Tail Phenomena: Probabilistic and Statistical Modeling
S. I. Resnick · 2007
Earlier work this paper cites.
The spectrum of heavy tailed random matrices
G. Ben Arous and A. Guionnet · 2008
Cited alongside, same era.
Learning with spectral kernels and heavy-tailed data
M. W. Mahoney and H. Narayanan · 2009
Cited alongside, same era.
Power-law distributions in empirical data
A. Clauset, C. R. Shalizi, and M. E. J. Newman · 2009
Cited alongside, same era.
Poisson convergence for the largest eigenvalues of heavy tailed random matrices
A. Auffinger, G. Ben Arous, and S. Péché · 2009
Cited alongside, same era.
Z. Burda and J. Jurkiewicz · 2009
Cited alongside, same era.
Random matrix theory and its innovative applications
Spectral norm regularization for improving the generalizability of deep learning
Y. Yoshida and T. Miyato · 2017
Later among the works it cites.
Generalization in deep learning
K. Kawaguchi, L. P. Kaelbling, and Y. Bengio · 2017
Later among the works it cites.
A PAC-Bayesian approach to spectrally-normalized margin bounds for neural networks
B. Neyshabur, S. Bhojanapalli, and N. Srebro · 2017
Later among the works it cites.
C. H. Martin and M. W. Mahoney · 2017
Later among the works it cites.
A surprising linear relationship predicts test performance in deep networks
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
A. Edelman and Y. Wang · 2013
Cited alongside, same era.
In search of the real inductive bias: on the role of implicit regularization in deep learning
B. Neyshabur, R. Tomioka, and N. Srebro · 2014
Cited alongside, same era.
powerlaw: A python package for analysis of heavy-tailed distributions
J. Alstott, E. Bullmore, and D. Plenz · 2014
Cited alongside, same era.
Limit theory for the largest eigenvalues of sample covariance matrices with heavy-tails
R. A. Davis, O. Pfaffel, and R. Stelzer · 2014
Cited alongside, same era.
Batch learning from logged bandit feedback through counterfactual risk minimization
A. Swaminathan and T. Joachims · 2015
Cited alongside, same era.
Norm-based capacity control in neural networks
B. Neyshabur, R. Tomioka, and N. Srebro · 2015
Cited alongside, same era.
Extreme eigenvalues of sparse, heavy tailed random matrices
A. Auffinger and S. Tang · 2016
Cited alongside, same era.
Q. Liao, B. Miranda, A. Banburski, J. Hidary, and T. Poggio · 2018
Later among the works it cites.
Theory IIIb: Generalization in deep networks
T. Poggio, Q. Liao, B. Miranda, A. Banburski, X. Boix, and J. Hidary · 2018
Later among the works it cites.
Stronger generalization bounds for deep nets via a compression approach
S. Arora, R. Ge, B. Neyshabur, and Y. Zhang · 2018
Later among the works it cites.
On the optimization of deep networks: Implicit acceleration by overparameterization
S. Arora, N. Cohen, and E. Hazan · 2018
Later among the works it cites.
Understanding generalization and optimization performance of deep CNNs
P. Zhou and J. Feng · 2018
Later among the works it cites.
C. H. Martin and M. W. Mahoney · 2018
Later among the works it cites.
Unpublished results, 2018
C. H. Martin and M. W. Mahoney · 2018
Later among the works it cites.
The jamming transition as a paradigm to understand the loss landscape of deep neural networks
M. Geiger, S. Spigler, S. d’Ascoli, L. Sagun, M. Baity-Jesi, G. Biroli, and M. Wyart · 2018
Later among the works it cites.
A jamming transition from under- to over-parametrization affects loss landscape and generalization
S. Spigler, M. Geiger, S. d’Ascoli, L. Sagun, G. Biroli, and M. Wyart · 2018
Later among the works it cites.
Traditional and heavy-tailed self regularization in neural network models
C. H. Martin and M. W. Mahoney · 2019
Closest in time.
Heavy-tailed Universality predicts trends in test accuracies for very large pre-trained deep neural networks
C. H. Martin and M. W. Mahoney · 2020
Closest in time.