2016

Robustly representing uncertainty in deep neural networks through sampling

McClure, Patrick, Kriegeskorte, Nikolaus

Understand

As deep neural networks (DNNs) are applied to increasingly challenging problems, they will need to be able to represent their own uncertainty.

Modeling uncertainty is one of the key features of Bayesian methods.
Using Bernoulli dropout with sampling at prediction time has recently been proposed as an efficient and well performing variational inference method for DNNs.
However, sampling from other multiplicative noise based variational distributions has not been investigated in depth.

Bayesian variable selection in linear regression
Toby J Mitchell and John J Beauchamp · 1988
Earlier work this paper cites.
A practical bayesian framework for backpropagation networks
David JC MacKay · 1992
Earlier work this paper cites.
Keeping the neural networks simple by minimizing the description length of the weights
Geoffrey E Hinton and Drew Van Camp · 1993
Earlier work this paper cites.
Model selection and accounting for model uncertainty in graphical models using occam’s window
David Madigan and Adrian E Raftery · 1994
Earlier work this paper cites.
Approaches for bayesian variable selection
Edward I George and Robert E McCulloch · 1997
Earlier work this paper cites.
Ensemble learning in bayesian neural networks
David Barber and Christopher M Bishop · 1998
Earlier work this paper cites.
Gradient-based learning applied to document recognition
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner · 1998
Earlier work this paper cites.
Spike and slab variable selection: frequentist and bayesian strategies
Hemant Ishwaran and J Sunil Rao · 2005
Earlier work this paper cites.
Learning multiple layers of features from tiny images, 2009
Alex Krizhevsky and Geoffrey Hinton · 2009
Earlier work this paper cites.

Practical variational inference for neural networks
Alex Graves · 2011
Cited alongside, same era.
Bayesian learning via stochastic gradient langevin dynamics
Max Welling and Yee W Teh · 2011
Cited alongside, same era.
ImageNet Classification with Deep Convolutional Neural Networks
Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton, Krizhevsky Alex, Ilya Sutskever, and Geoffrey E Hinton · 2012
Cited alongside, same era.
Bayesian learning for neural networks , volume 118
Radford M Neal · 2012
Cited alongside, same era.
Regularization of neural networks using dropconnect
Li Wan, Matthew Zeiler, Sixin Zhang, Yann L Cun, and Rob Fergus · 2013
Cited alongside, same era.
Expectation propagation for neural networks with sparsity-promoting priors
Pasi Jylänki, Aapo Nummenmaa, and Aki Vehtari · 2014
Cited alongside, same era.
Dropout: A simple way to prevent neural networks from overfitting
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov · 2014
Cited alongside, same era.
Weight uncertainty in neural network
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra · 2015
Cited alongside, same era.
Dropout as a bayesian approximation: Insights and applications
Yarin Gal and Zoubin Ghahramani · 2015
Cited alongside, same era.

Probabilistic backpropagation for scalable learning of bayesian neural networks
José Miguel Hernández-Lobato and Ryan Adams · 2015
Later among the works it cites.
Variational dropout and the local reparameterization trick
Diederik P Kingma, Tim Salimans, and Max Welling · 2015
Later among the works it cites.
Smart regularization of deep architectures
Christos Louizos · 2015
Later among the works it cites.
Human-level control through deep reinforcement learning
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei a Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis · 2015
Later among the works it cites.
Uncertainty in Deep Learning
Yarin Gal · 2016
Closest in time.
Bayesian convolutional neural networks with Bernoulli approximate variational inference
Yarin Gal and Zoubin Ghahramani · 2016
Closest in time.
Mastering the game of go with deep neural networks and tree search
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al · 2016
Closest in time.
What uncertainties do we need in bayesian deep learning for computer vision?
Original
Alex Kendall and Yarin Gal · 2017
Closest in time.

alphaXiv searches the wider corpus for related work and actual follow-ups.

alphaXiv is searching for related work…

Reading the bibliography…