2020

How Good is the Bayes Posterior in Deep Neural Networks Really?

Wenzel, Florian, Roth, Kevin, Veeling, Bastiaan S. et al.

Understand

During the past five years the Bayesian deep learning community has developed increasingly accurate and efficient approximate inference procedures that allow for Bayesian inference in deep neural networks.

However, despite this algorithmic progress and the promise of improved uncertainty quantification and sample efficiency there are---as of early 2020---no publicized deployments of Bayesian neural networks in industrial practice.
In this work we cast doubt on the current understanding of Bayes posteriors in popular deep neural networks: we demonstrate through careful MCMC sampling that the posterior predictive induced by the Bayes posterior yields systematically worse predictions compared to simpler methods including point estimates obtained from SGD.
Furthermore, we demonstrate that predictive performance is improved significantly through the use of a "cold posterior" that overcounts evidence.

Built on

Sur la théorie du mouvement brownien
Langevin, P · 1908
Earlier work this paper cites.
Verification of forecasts expressed in terms of probability
Brier, G. W · 1950
Earlier work this paper cites.
On bias reduction in estimation
Schucany, W., Gray, H., and Owen, D · 1971
Earlier work this paper cites.
Statistical decision theory and Bayesian analysis
Berger, J. O · 1985
Earlier work this paper cites.
Replica Monte Carlo simulation of spin-glasses
Swendsen, R. H. and Wang, J.-S · 1986
Earlier work this paper cites.
Hybrid Monte Carlo
Duane, S., Kennedy, A. D., Pendleton, B. J., and Roweth, D · 1987
Earlier work this paper cites.
Inference from iterative simulation using multiple sequences
Gelman, A. and Rubin, D. B · 1992
Earlier work this paper cites.
An Introduction to Predictive Inference
Geisser, S · 1993
Earlier work this paper cites.
Keeping neural networks simple by minimizing the description length of the weights
Hinton, G. and Van Camp, D · 1993
Earlier work this paper cites.
Ensemble learning and evidence maximization
MacKay, D. J. et al · 1995
Earlier work this paper cites.
Bayesian learning for neural networks
Neal, R. M · 1995
Earlier work this paper cites.
Estimating functions of probability distributions from a finite set of samples
Wolpert, D. H. and Wolf, D. R · 1995
Earlier work this paper cites.
On asymptotic properties of predictive distributions
Komaki, F · 1996
Earlier work this paper cites.
Ensemble learning for multi-layer networks
Barber, D. and Bishop, C. M · 1998
Earlier work this paper cites.
Replica-exchange molecular dynamics method for protein folding
Sugita, Y. and Okamoto, Y · 1999
Earlier work this paper cites.
Entropy and inference, revisited
Nemenman, I., Shafee, F., and Bialek, W · 2002
Earlier work this paper cites.
On the Markov chain central limit theorem
Jones, G. L. et al · 2004
Earlier work this paper cites.
Parallel tempering: Theory, applications, and new perspectives
Earl, D. J. and Deem, M. W · 2005
Earlier work this paper cites.
Bootstrap prediction and Bayesian prediction under misspecified models
Fushiki, T. et al · 2005
Earlier work this paper cites.
On variance conditions for markov chain clts
Häggström, O. and Rosenthal, J · 2007
Earlier work this paper cites.
Learning multiple layers of features from tiny images
Krizhevsky, A., Hinton, G., et al · 2009
Earlier work this paper cites.
Understanding the difficulty of training deep feedforward neural networks
Glorot, X. and Bengio, Y · 2010
Earlier work this paper cites.
Handbook of Markov Chain Monte Carlo
Brooks, S., Gelman, A., Jones, G., and Meng, X · 2011
Earlier work this paper cites.
MCMC using Hamiltonian dynamics
Neal, R. M. et al · 2011
Earlier work this paper cites.
Bayesian learning via stochastic gradient Langevin dynamics
Welling, M. and Teh, Y. W · 2011
Earlier work this paper cites.
Machine learning: a probabilistic perspective
Murphy, K. P · 2012
Earlier work this paper cites.
Lecture 6.5—RmsProp: Divide the gradient by a running average of its recent magnitude
Tieleman, T. and Hinton, G · 2012
Earlier work this paper cites.
Bayesian data analysis
Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A., and Rubin, D. B · 2013
Earlier work this paper cites.
Robust Bayesian inference under model misspecification, 2013
Jansen, L · 2013
Earlier work this paper cites.
On the importance of initialization and momentum in deep learning
Sutskever, I., Martens, J., Dahl, G., and Hinton, G · 2013
Earlier work this paper cites.
Stochastic gradient Hamiltonian Monte Carlo
Chen, T., Fox, E., and Guestrin, C · 2014
Earlier work this paper cites.
Bayesian sampling using stochastic gradient thermostats
Ding, N., Fang, Y., Babbush, R., Chen, C., Skeel, R. D., and Neven, H · 2014
Earlier work this paper cites.
The no-u-turn sampler: adaptively setting path lengths in Hamiltonian Monte Carlo
Hoffman, M. D. and Gelman, A · 2014
Earlier work this paper cites.

Similar

Dropout: a simple way to prevent neural networks from overfitting
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R · 2014
Cited alongside, same era.
Hamiltonian Monte Carlo for hierarchical models
Betancourt, M. and Girolami, M · 2015
Cited alongside, same era.
Weight uncertainty in neural network
Blundell, C., Cornebise, J., Kavukcuoglu, K., and Wierstra, D · 2015
Cited alongside, same era.
On the convergence of stochastic gradient MCMC algorithms with high-order integrators
Chen, C., Ding, N., and Carin, L · 2015
Cited alongside, same era.
Delving deep into rectifiers: Surpassing human-level performance on imagenet classification
He, K., Zhang, X., Ren, S., and Sun, J · 2015
Cited alongside, same era.
Batch normalization: Accelerating deep network training by reducing internal covariate shift
Original
Ioffe, S. and Szegedy, C · 2015
Cited alongside, same era.
Variational dropout and the local reparameterization trick
Kingma, D. P., Salimans, T., and Welling, M · 2015
Cited alongside, same era.
A complete recipe for stochastic gradient MCMC
Ma, Y.-A., Chen, T., and Fox, E · 2015
Cited alongside, same era.
Obtaining well calibrated probabilities using bayesian binning
Naeini, M. P., Cooper, G., and Hauskrecht, M · 2015
Cited alongside, same era.
On posterior concentration in misspecified models
Ramamoorthi, R. V., Sriram, K., and Martin, R · 2015
Cited alongside, same era.
Covariance-controlled adaptive Langevin thermostat for large-scale Bayesian sampling
Shang, X., Zhu, Z., Leimkuhler, B., and Storkey, A. J · 2015
Cited alongside, same era.
Tensorflow: A system for large-scale machine learning
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al · 2016
Cited alongside, same era.

Then

Deep neural networks as gaussian processes
Lee, J., Bahri, Y., Novak, R., Schoenholz, S. S., Pennington, J., and Sohl-Dickstein, J · 2018
Later among the works it cites.
Revisiting small batch training for deep neural networks
Original
Masters, D. and Luschi, C · 2018
Later among the works it cites.
Debiasing evidence approximations: On importance-weighted autoencoders and jackknife variational inference
Nowozin, S · 2018
Later among the works it cites.
Stochastic normalizations as Bayesian learning
Original
Shekhovtsov, A. and Flach, B · 2018
Later among the works it cites.
Fluctuation-dissipation relations for stochastic gradient descent
Original
Yaida, S · 2018
Later among the works it cites.
Noisy natural gradient as variational inference
Zhang, G., Sun, S., Duvenaud, D., and Grosse, R · 2018
Later among the works it cites.
Bayesian neural networks at finite temperature
Original
Baldock, R. J. and Marzari, N · 2019
Later among the works it cites.
Bayesian fractional posteriors
Bhattacharya, A., Pati, D., Yang, Y., et al · 2019
Later among the works it cites.
Deep convolutional networks as shallow gaussian processes
Garriga-Alonso, A., Rasmussen, C. E., and Aitchison, L · 2019
Later among the works it cites.
Tati-thermodynamic analytics toolkit: Tensorflow-based software for posterior sampling in machine learning applications
Original
Heber, F., Trstanova, Z., and Leimkuhler, B · 2019
Later among the works it cites.
Multi-sample dropout for accelerated training and better generalization
Original
Inoue, H · 2019
Later among the works it cites.
Partitioned integrators for thermodynamic parameterization of neural networks
Original
Leimkuhler, B., Matthews, C., and Vlaar, T · 2019
Later among the works it cites.
Sharpening jensen’s inequality
Liao, J. and Berg, A · 2019
Later among the works it cites.
Learning under model misspecification: Applications to variational and ensemble methods
Masegosa, A. R · 2019
Later among the works it cites.
Dropout as a structured shrinkage prior
Nalisnick, E., Hernandez-Lobato, J. M., and Smyth, P · 2019
Later among the works it cites.
Bayesian deep convolutional networks with many channels are gaussian processes
Novak, R., Xiao, L., Bahri, Y., Lee, J., Yang, G., Hron, J., Abolafia, D. A., Pennington, J., and Sohl-Dickstein, J · 2019
Later among the works it cites.
Practical deep learning with Bayesian principles
Original
Osawa, K., Swaroop, S., Jain, A., Eschenhagen, R., Turner, R. E., Yokota, R., and Khan, M. E · 2019
Later among the works it cites.
Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift
Ovadia, Y., Fertig, E., Ren, J., Nado, Z., Sculley, D., Nowozin, S., Dillon, J. V., Lakshminarayanan, B., and Snoek, J · 2019
Later among the works it cites.
Expressive priors in bayesian neural networks: Kernel combinations and periodic functions
Original
Pearce, T., Zaki, M., Brintrup, A., and Neely, A · 2019
Later among the works it cites.
Applied stochastic differential equations , volume 10
Särkkä, S. and Solin, A · 2019
Later among the works it cites.
A tail-index analysis of stochastic gradient noise in deep neural networks
Original
Simsekli, U., Sagun, L., and Gurbuzbalaban, M · 2019
Later among the works it cites.
Filter response normalization layer: Eliminating batch dependence in the training of deep neural networks
Original
Singh, S. and Krishnan, S · 2019
Later among the works it cites.
Functional variational Bayesian neural networks
Original
Sun, S., Zhang, G., Shi, J., and Grosse, R · 2019
Later among the works it cites.
BatchEnsemble: Efficient ensemble of deep neural networks via rank-1 perturbation
Wen, Y., Tran, D., and Ba, J · 2019
Later among the works it cites.
The case for Bayesian deep learning
Wilson, A. G · 2019
Later among the works it cites.
Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural tangent kernel derivation
Original
Yang, G · 2019
Later among the works it cites.
The anisotropic noise in stochastic gradient descent: Its behavior of escaping from sharp minima and regularization effects
Zhu, Z., Wu, J., Yu, B., Wu, L., and Ma, J · 2019
Later among the works it cites.
Pitfalls of in-domain uncertainty estimation and ensembling in deep learning
Ashukha, A., Lyzhov, A., Molchanov, D., and Vetrov, D · 2020
Closest in time.
Bayesian inference for large scale image classification
Heek, J. and Kalchbrenner, N · 2020
Closest in time.
tfp.mcmc: Modern Markov chain Monte Carlo tools built for modern hardware, 2020
Lao, J., Suter, C., Langmore, I., Chimisov, C., Saxena, A., Sountsov, P., Moore, D., Saurous, R. A., Hoffman, M. D., and Dillon, J. V · 2020
Closest in time.
Bayesian deep learning and a probabilistic perspective of generalization
Original
Wilson, A. G. and Izmailov, P · 2020
Closest in time.
Cyclical stochastic gradient MCMC for Bayesian deep learning
Zhang, R., Li, C., Zhang, J., Chen, C., and Wilson, A. G · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…