2020

Predicting Neural Network Accuracy from Weights

Unterthiner, Thomas, Keysers, Daniel, Gelly, Sylvain et al.

Understand

We show experimentally that the accuracy of a trained neural network can be predicted surprisingly well by looking only at its weights, without evaluating it on input data.

We motivate this task and introduce a formal setting for it.
Even when using simple statistics of the weights, the predictors are able to rank neural networks by their performance with very high accuracy (R2 score more than 0.98).
Furthermore, the predictors are able to rank networks trained on different, unobserved datasets and with different architectures.

Built on

Learning effective loss functions efficiently
Streeter, M · 1907
Earlier work this paper cites.
Statistical Learning Theory
Vapnik, V · 1998
Earlier work this paper cites.
Introduction to Nonparametric Estimation
Tsybakov, A. B · 2008
Earlier work this paper cites.
Learning multiple layers of features from tiny images
Krizhevsky, A · 2009
Earlier work this paper cites.
Understanding the difficulty of training deep feedforward neural networks
Glorot, X. and Bengio, Y · 2010
Earlier work this paper cites.
MNIST handwritten digit database
LeCun, Y., Cortes, C., and Burges, C · 2010
Earlier work this paper cites.
Reading digits in natural images with unsupervised feature learning
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., and Ng, A. Y · 2011
Earlier work this paper cites.
Adam: A method for stochastic optimization
Kingma, D. P. and Lei, J · 2014
Earlier work this paper cites.
Network in network
Lin, M., Chen, Q., and Yan, S · 2014
Earlier work this paper cites.
Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
Saxe, A., McClelland, J., and Ganguli, S · 2014
Earlier work this paper cites.
Understanding machine learning: from theory to algorithms
Shalev-Shwartz, S. and Ben-David, S · 2014
Earlier work this paper cites.
Freeze-thaw Bayesian optimization
Swersky, K., Snoek, J., and Adams, R. P · 2014
Earlier work this paper cites.
Speeding up automatic hyperparameter optimization of deep neural networks by extrapolation of learning curves
Domhan, T., Springenberg, J. T., and Hutter, F · 2015
Earlier work this paper cites.

Similar

Delving deep into rectifiers: Surpassing human-level performance on imagenet classification
He, K., Zhang, X., Ren, S., and Sun, J · 2015
Cited alongside, same era.
Norm-based capacity control in neural networks
Neyshabur, B., Tomioka, R., and Srebro, N · 2015
Cited alongside, same era.
Striving for simplicity: The all convolutional net
Springenberg, J., Dosovitskiy, A., Brox, T., and Riedmiller, M · 2015
Cited alongside, same era.
Identity mappings in deep residual networks
He, K., Zhang, X., Ren, S., and Sun, J · 2016
Cited alongside, same era.
Accelerating neural architecture search using performance prediction
Baker, B., Gupta, O., Raskar, R., and Naik, N · 2017
Cited alongside, same era.
Spectrally-normalized margin bounds for neural networks
Bartlett, P. L., Foster, D. J., and Telgarsky, M. J · 2017
Cited alongside, same era.
LightGBM: A highly efficient gradient boosting decision tree
Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., and Liu, T.-Y · 2017
Cited alongside, same era.
Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms
Xiao, H., Rasul, K., and Vollgraf, R · 2017
Cited alongside, same era.
Deep sets
Zaheer, M., Kottur, S., Ravanbakhsh, S., Poczos, B., Salakhutdinov, R. R., and Smola, A. J · 2017
Cited alongside, same era.
Understanding deep learning requires rethinking generalization
Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O · 2017
Cited alongside, same era.
Reconciling modern machine learning and the bias-variance trade-off
Belkin, M., Hsu, D., Ma, S., and Mandal, S · 2018
Cited alongside, same era.
Large margin deep networks for classification
Elsayed, G. F., Krishnan, K., Mobahi, H., Regan, K., and Bengio, S · 2018
Cited alongside, same era.

Then

Implicit self-regularization in deep neural networks: Evidence from random matrix theory and implications for learning
Martin, C. H. and Mahoney, M. W · 2018
Later among the works it cites.
Predicting the accuracy of neural networks from final and intermediate layer outputs
DeChant, C., Han, S., and Lipson, H · 2019
Later among the works it cites.
The lottery ticket hypothesis: Finding sparse, trainable neural networks
Frankle, J. and Carbin, M · 2019
Later among the works it cites.
TAPAS: Train-less accuracy predictor for architecture search
Istrate, R., Scheidegger, F., Mariani, G., Nikolopoulos, D., Bekas, C., and Malossi, A. C. I · 2019
Later among the works it cites.
Predicting the generalization gap in deep networks with margin distributions
Jiang, Y., Krishnan, D., Mobahi, H., and Bengio, S · 2019
Later among the works it cites.
One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers
Morcos, A., Yu, H., Paganini, M., and Tian, Y · 2019
Later among the works it cites.
Towards task and architecture-independent generalization gap predictors
Yak, S., Gonzalvo, J., and Mazzawi, H · 2019
Later among the works it cites.
Are all layers created equal?
Zhang, C., Bengio, S., and Singer, Y · 2019
Later among the works it cites.
Classifying the classifier: dissecting the weight space of neural networks
Eilertsen, G., Jönsson, D., Ropinski, T., Unger, J., and Ynnerman, A · 2020
Closest in time.
Fantastic generalization measures and where to find them
Jiang, Y., Neyshabur, B., Krishnan, D., Mobahi, H., and Bengio, S · 2020
Closest in time.
Heavy-tailed universality predicts trends in test accuracies for very large pre-trained deep neural networks
Martin, C. H. and Mahoney, M. W · 2020
Closest in time.
Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data
Martin, C. H., Tongsu, Peng, and Mahoney, M. W · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…