2019

On the Expressiveness of Approximate Inference in Bayesian Neural Networks

Foong, Andrew Y. K., Burt, David R., Li, Yingzhen et al.

Understand

While Bayesian neural networks (BNNs) hold the promise of being flexible, well-calibrated statistical models, inference often requires approximations whose consequences are poorly understood.

We study the quality of common variational methods in approximating the Bayesian predictive distribution.
For single-hidden layer ReLU BNNs, we prove a fundamental limitation in function-space of two of the most commonly used distributions defined in weight-space: mean-field Gaussian and Monte Carlo dropout.
We find there are simple cases where neither method can have substantially increased uncertainty in between well-separated regions of low uncertainty.

Built on

Handbook of mathematical functions: with formulas, graphs, and mathematical tables , volume 55
Milton Abramowitz and Irene A Stegun · 1965
Earlier work this paper cites.
Transforming neural-net output levels to probability distributions
John S Denker and Yann Lecun · 1991
Earlier work this paper cites.
A practical Bayesian framework for backpropagation networks
David J. C. MacKay · 1992
Earlier work this paper cites.
Keeping the neural networks simple by minimizing the description length of the weights
Geoffrey E Hinton and Drew Van Camp · 1993
Earlier work this paper cites.
Multilayer feedforward networks with a nonpolynomial activation function can approximate any function
Moshe Leshno, Vladimir Ya Lin, Allan Pinkus, and Shimon Schocken · 1993
Earlier work this paper cites.
Bayesian learning for neural networks
Radford M Neal · 1995
Earlier work this paper cites.
Ensemble learning in Bayesian neural networks
David Barber and Christopher M Bishop · 1998
Earlier work this paper cites.
Variational learning in nonlinear Gaussian belief networks
Brendan J Frey and Geoffrey E Hinton · 1999
Earlier work this paper cites.
An introduction to variational methods for graphical models
Michael I Jordan, Zoubin Ghahramani, Tommi S Jaakkola, and Lawrence K Saul · 1999
Earlier work this paper cites.
Variational algorithms for approximate Bayesian inference
Matthew James Beal · 2003
Earlier work this paper cites.
Visualizing data using t-SNE
Laurens van der Maaten and Geoffrey Hinton · 2008
Earlier work this paper cites.
Active learning literature survey
Burr Settles · 2009
Earlier work this paper cites.
MCMC using Hamiltonian dynamics
Radford M Neal et al · 2011
Earlier work this paper cites.
Bayesian learning via stochastic gradient Langevin dynamics
Max Welling and Yee W Teh · 2011
Earlier work this paper cites.
Stochastic gradient Hamiltonian Monte Carlo
Tianqi Chen, Emily Fox, and Carlos Guestrin · 2014
Earlier work this paper cites.
Machine learning approaches for improving condition-based maintenance of naval propulsion plants
Andrea Coraddu, Luca Oneto, Alessandro Ghio, Stefano Savio, Davide Anguita, and Massimo Figari · 2014
Earlier work this paper cites.

Similar

The No-U-Turn sampler: adaptively setting path lengths in Hamiltonian Monte Carlo
Matthew D Hoffman and Andrew Gelman · 2014
Cited alongside, same era.
Weight uncertainty in neural networks
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra · 2015
Cited alongside, same era.
Probabilistic backpropagation for scalable learning of Bayesian neural networks
José Miguel Hernández-Lobato and Ryan Adams · 2015
Cited alongside, same era.
Variational dropout and the local reparameterization trick
Durk P Kingma, Tim Salimans, and Max Welling · 2015
Cited alongside, same era.
Stochastic expectation propagation
Yingzhen Li, José Miguel Hernández-Lobato, and Richard E Turner · 2015
Cited alongside, same era.
Uncertainty in deep learning
Yarin Gal · 2016
Cited alongside, same era.
Dropout as a Bayesian approximation: Representing model uncertainty in deep learning
Yarin Gal and Zoubin Ghahramani · 2016
Cited alongside, same era.
Black-box α \alpha -divergence minimization
José Miguel Hernández-Lobato, Yingzhen Li, Mark Rowland, Thang Bui, Daniel Hernández-Lobato, and Richard Turner · 2016
Cited alongside, same era.
Rényi divergence variational inference
Yingzhen Li and Richard E Turner · 2016
Cited alongside, same era.
Variational inference: A review for statisticians
David M Blei, Alp Kucukelbir, and Jon D McAuliffe · 2017
Cited alongside, same era.
Deep Bayesian active learning with image data
Yarin Gal, Riashat Islam, and Zoubin Ghahramani · 2017
Cited alongside, same era.
Multiplicative normalizing flows for variational Bayesian neural networks
Christos Louizos and Max Welling · 2017
Cited alongside, same era.

Then

Fast and scalable Bayesian deep learning by weight-perturbation in Adam
Mohammad Emtiyaz Khan, Didrik Nielsen, Voot Tangkaratt, Wu Lin, Yarin Gal, and Akash Srivastava · 2018
Later among the works it cites.
Deep neural networks as Gaussian processes
Jaehoon Lee, Jascha Sohl-Dickstein, Jeffrey Pennington, Roman Novak, Sam Schoenholz, and Yasaman Bahri · 2018
Later among the works it cites.
Gaussian process behaviour in wide deep neural networks
Alexander G. de G. Matthews, Jiri Hron, Mark Rowland, Richard E. Turner, and Zoubin Ghahramani · 2018
Later among the works it cites.
On the importance of strong baselines in Bayesian deep learning
Original
Jishnu Mukhoti, Pontus Stenetorp, and Yarin Gal · 2018
Later among the works it cites.
Randomized prior functions for deep reinforcement learning
Ian Osband, John Aslanides, and Albin Cassirer · 2018
Later among the works it cites.
A scalable Laplace approximation for neural networks
Hippolyt Ritter, Aleksandar Botev, and David Barber · 2018
Later among the works it cites.
Neural network ensembles and variational inference revisited
Marcin B Tomczak, Siddharth Swaroop, and Richard E Turner · 2018
Later among the works it cites.
Benchmarking Bayesian deep learning with diabetic retinopathy diagnosis
Angelos Filos, Sebastian Farquhar, Aidan N. Gomez, Tim G. J. Rudner, Zachary Kenton, Lewis Smith, Milad Alizadeh, Arnoud de Kroon, and Yarin Gal · 2019
Closest in time.
Can you trust your model’s uncertainty? Evaluating predictive uncertainty under dataset shift
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D Sculley, Sebastian Nowozin, Joshua V Dillon, Balaji Lakshminarayanan, and Jasper Snoek · 2019
Closest in time.
Functional variational Bayesian neural networks
Shengyang Sun, Guodong Zhang, Jiaxin Shi, and Roger Grosse · 2019
Closest in time.
Improving and understanding variational continual learning
Original
Siddharth Swaroop, Cuong V Nguyen, Thang D Bui, and Richard E Turner · 2019
Closest in time.
Liberty or depth: Deep Bayesian neural nets do not need complex weight posterior approximations
Original
Sebastian Farquhar, Lewis Smith, and Yarin Gal · 2020
Closest in time.
Exact posterior distributions of wide Bayesian neural networks
Jiri Hron, Yasaman Bahri, Roman Novak, Jeffrey Pennington, and Jascha Sohl-Dickstein · 2020
Closest in time.
How good is the Bayes posterior in deep neural networks really?
Florian Wenzel, Kevin Roth, Bastiaan S Veeling, Jakub Świątkowski, Linh Tran, Stephan Mandt, Jasper Snoek, Tim Salimans, Rodolphe Jenatton, and Sebastian Nowozin · 2020
Closest in time.
Cyclical stochastic gradient MCMC for Bayesian deep learning
Ruqi Zhang, Chunyuan Li, Jianyi Zhang, Changyou Chen, and Andrew Gordon Wilson · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…