2021

Learning Neural Network Subspaces

Wortsman, Mitchell, Horton, Maxwell, Guestrin, Carlos et al.

Understand

Recent observations have advanced our understanding of the neural network optimization landscape, revealing the existence of (1) paths of high accuracy containing diverse solutions and (2) wider minima offering improved performance.

Previous methods observing diverse paths require multiple training runs.
In contrast we aim to leverage both property (1) and (2) with a single method and in a single training run.
With a similar computational cost as training one model, we learn lines, curves, and simplexes of high-accuracy neural networks.

Built on

The mnist database of handwritten digits
LeCun, Y · 1998
Earlier work this paper cites.
Imagenet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L · 2009
Earlier work this paper cites.
Learning multiple layers of features from tiny images
Krizhevsky, A., Hinton, G., et al · 2009
Earlier work this paper cites.
Bayesian learning via stochastic gradient langevin dynamics
Welling, M. and Teh, Y. W · 2011
Earlier work this paper cites.
Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
Dauphin, Y. N., Pascanu, R., Gulcehre, C., Cho, K., Ganguli, S., and Bengio, Y · 2014
Earlier work this paper cites.
Dropout: a simple way to prevent neural networks from overfitting
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R · 2014
Earlier work this paper cites.
Weight uncertainty in neural networks
Original
Blundell, C., Cornebise, J., Kavukcuoglu, K., and Wierstra, D · 2015
Earlier work this paper cites.
Mxnet: A flexible and efficient machine learning library for heterogeneous distributed systems
Original
Chen, T., Li, M., Li, Y., Lin, M., Wang, N., Wang, M., Xiao, T., Xu, B., Zhang, C., and Zhang, Z · 2015
Earlier work this paper cites.
Delving deep into rectifiers: Surpassing human-level performance on imagenet classification
He, K., Zhang, X., Ren, S., and Sun, J · 2015
Earlier work this paper cites.
Batch normalization: Accelerating deep network training by reducing internal covariate shift
Ioffe, S. and Szegedy, C · 2015
Earlier work this paper cites.
Tiny imagenet visual recognition challenge
Le, Y. and Yang, X · 2015
Earlier work this paper cites.
Layer normalization
Original
Ba, J. L., Kiros, J. R., and Hinton, G. E · 2016
Earlier work this paper cites.
Dropout as a bayesian approximation: Representing model uncertainty in deep learning
Gal, Y. and Ghahramani, Z · 2016
Earlier work this paper cites.
Deep residual learning for image recognition
He, K., Zhang, X., Ren, S., and Sun, J · 2016
Earlier work this paper cites.
Sgdr: Stochastic gradient descent with warm restarts
Original
Loshchilov, I. and Hutter, F · 2016
Earlier work this paper cites.

Similar

Wide residual networks
Original
Zagoruyko, S. and Komodakis, N · 2016
Cited alongside, same era.
On calibration of modern neural networks
Original
Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q · 2017
Cited alongside, same era.
Snapshot ensembles: Train 1, get m for free
Original
Huang, G., Li, Y., Pleiss, G., Liu, Z., Hopcroft, J. E., and Weinberger, K. Q · 2017
Cited alongside, same era.
Simple and scalable predictive uncertainty estimation using deep ensembles
Lakshminarayanan, B., Pritzel, A., and Blundell, C · 2017
Cited alongside, same era.
Essentially no barriers in neural network energy landscape
Original
Draxler, F., Veschgini, K., Salmhofer, M., and Hamprecht, F. A · 2018
Cited alongside, same era.
Entropy-SGD optimizes the prior of a PAC-Bayes bound: Generalization properties of entropy-SGD and data-dependent priors
Dziugaite, G. K. and Roy, D · 2018
Cited alongside, same era.
Loss surfaces, mode connectivity, and fast ensembling of dnns
Garipov, T., Izmailov, P., Podoprikhin, D., Vetrov, D. P., and Wilson, A. G · 2018
Cited alongside, same era.
Averaging weights leads to wider optima and better generalization
Original
Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D., and Wilson, A. G · 2018
Cited alongside, same era.
Group normalization
Wu, Y. and He, K · 2018
Cited alongside, same era.
Entropy-sgd: Biasing gradient descent into wide valleys
Chaudhari, P., Choromanska, A., Soatto, S., LeCun, Y., Baldassi, C., Borgs, C., Chayes, J., Sagun, L., and Zecchina, R · 2019
Cited alongside, same era.
The difficulty of training sparse neural networks
Original
Evci, U., Pedregosa, F., Gomez, A., and Elsen, E · 2019
Cited alongside, same era.
Large scale structure of neural network loss landscapes
Fort, S. and Jastrzebski, S · 2019
Cited alongside, same era.

Then

When does label smoothing help?
Original
Müller, R., Kornblith, S., and Hinton, G · 2019
Later among the works it cites.
Pytorch: An imperative style, high-performance deep learning library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S · 2019
Later among the works it cites.
Exploring randomly wired neural networks for image recognition
Xie, S., Kirillov, A., Girshick, R., and He, K · 2019
Later among the works it cites.
Underspecification presents challenges for credibility in modern machine learning
Original
D’Amour, A., Heller, K., Moldovan, D., Adlam, B., Alipanahi, B., Beutel, A., Chen, C., Deaton, J., Eisenstein, J., Hoffman, M. D., et al · 2020
Later among the works it cites.
Sharpness-aware minimization for efficiently improving generalization
Original
Foret, P., Kleiner, A., Mobahi, H., and Neyshabur, B · 2020
Later among the works it cites.
Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the neural tangent kernel
Original
Fort, S., Dziugaite, G. K., Paul, M., Kharaghani, S., Roy, D. M., and Ganguli, S · 2020
Later among the works it cites.
Revisiting” qualitatively characterizing neural network optimization problems”
Original
Frankle, J · 2020
Later among the works it cites.
Linear mode connectivity and the lottery ticket hypothesis
Frankle, J., Dziugaite, G. K., Roy, D., and Carbin, M · 2020
Later among the works it cites.
Subspace inference for bayesian deep learning
Izmailov, P., Maddox, W. J., Kirichenko, P., Garipov, T., Vetrov, D., and Wilson, A. G · 2020
Later among the works it cites.
Pruning neural networks without any data by iteratively conserving synaptic flow
Original
Tanaka, H., Kunin, D., Yamins, D. L., and Ganguli, S · 2020
Later among the works it cites.
Measuring robustness to natural distribution shifts in image classification
Taori, R., Dave, A., Shankar, V., Carlini, N., Recht, B., and Schmidt, L · 2020
Later among the works it cites.
Batchensemble: an alternative approach to efficient ensemble and lifelong learning
Original
Wen, Y., Tran, D., and Ba, J · 2020
Later among the works it cites.
Cyclical stochastic gradient mcmc for bayesian deep learning
Zhang, R., Li, C., Zhang, J., Chen, C., and Wilson, A. G · 2020
Later among the works it cites.
Neural networks with late-phase weights
Oswald, J. V., Kobayashi, S., Sacramento, J., Meulemans, A., Henning, C., and Grewe, B. F · 2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…