2020

Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning

Ashukha, Arsenii, Lyzhov, Alexander, Molchanov, Dmitry et al.

Understand

Uncertainty estimation and ensembling methods go hand-in-hand.

Uncertainty estimation is one of the main benchmarks for assessment of ensembling performance.
At the same time, deep learning ensembles have provided state-of-the-art results in uncertainty estimation.
In this work, we focus on in-domain uncertainty for image classification.

Built on

Verification of forecasts expressed in terms of probability
Glenn W Brier · 1950
Earlier work this paper cites.
Neural network ensembles
Lars Kai Hansen and Peter Salamon · 1990
Earlier work this paper cites.
Evaluating predictive uncertainty challenge
Joaquin Quinonero-Candela, Carl Edward Rasmussen, Fabian Sinz, Olivier Bousquet, and Bernhard Schölkopf · 2005
Earlier work this paper cites.
Learning multiple layers of features from tiny images
Alex Krizhevsky, Geoffrey Hinton, et al · 2009
Earlier work this paper cites.
Accuracy-rejection curves (arcs) for comparing classification methods with a reject option
Malik Sajjad Ahmed Nadeem, Jean-Daniel Zucker, and Blaise Hanczar · 2009
Earlier work this paper cites.
Bayesian learning via stochastic gradient langevin dynamics
Max Welling and Yee W Teh · 2011
Earlier work this paper cites.
Active learning
Burr Settles · 2012
Earlier work this paper cites.
Intriguing properties of neural networks
Original
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus · 2013
Earlier work this paper cites.
Fast dropout training
Sida Wang and Christopher Manning · 2013
Earlier work this paper cites.
Very deep convolutional networks for large-scale image recognition
Original
Karen Simonyan and Andrew Zisserman · 2014
Earlier work this paper cites.
Dropout: a simple way to prevent neural networks from overfitting
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov · 2014
Earlier work this paper cites.
Bayesian dark knowledge
Anoop Korattikara Balan, Vivek Rathod, Kevin P Murphy, and Max Welling · 2015
Earlier work this paper cites.
Weight uncertainty in neural networks
Original
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra · 2015
Earlier work this paper cites.
Distilling the knowledge in a neural network
Original
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean · 2015
Earlier work this paper cites.
Batch normalization: Accelerating deep network training by reducing internal covariate shift
Original
Sergey Ioffe and Christian Szegedy · 2015
Earlier work this paper cites.
Variational dropout and the local reparameterization trick
Durk P Kingma, Tim Salimans, and Max Welling · 2015
Earlier work this paper cites.
Optimizing neural networks with kronecker-factored approximate curvature
James Martens and Roger Grosse · 2015
Earlier work this paper cites.
Obtaining well calibrated probabilities using bayesian binning
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht · 2015
Earlier work this paper cites.
Imagenet large scale visual recognition challenge
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al · 2015
Earlier work this paper cites.

Similar

Going deeper with convolutions
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich · 2015
Cited alongside, same era.
92.45 on cifar-10 in torch, 2015
Sergey Zagoruyko · 2015
Cited alongside, same era.
Uncertainty in deep learning
Yarin Gal · 2016
Cited alongside, same era.
Dropout as a bayesian approximation: Representing model uncertainty in deep learning
Yarin Gal and Zoubin Ghahramani · 2016
Cited alongside, same era.
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2016
Cited alongside, same era.
How to train deep variational autoencoders and probabilistic ladder networks
Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, and Ole Winther · 2016
Cited alongside, same era.
Wide residual networks
Original
Sergey Zagoruyko and Nikos Komodakis · 2016
Cited alongside, same era.
On calibration of modern neural networks
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger · 2017
Cited alongside, same era.
Snapshot ensembles: Train 1, get m for free
Original
Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John E Hopcroft, and Kilian Q Weinberger · 2017
Cited alongside, same era.
Simple and scalable predictive uncertainty estimation using deep ensembles
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell · 2017
Cited alongside, same era.
Multiplicative normalizing flows for variational bayesian neural networks
Christos Louizos and Max Welling · 2017
Cited alongside, same era.
Variational dropout sparsifies deep neural networks
Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov · 2017
Cited alongside, same era.

Then

Predictive uncertainty estimation via prior networks
Andrey Malinin and Mark Gales · 2018
Later among the works it cites.
Inhibited softmax for uncertainty estimation in neural networks
Original
Marcin Możejko, Mateusz Susik, and Rafał Karczewski · 2018
Later among the works it cites.
A scalable laplace approximation for neural networks
Hippolyt Ritter, Aleksandar Botev, and David Barber · 2018
Later among the works it cites.
Bayesian uncertainty estimation for batch normalized deep networks
Mattias Teye, Hossein Azizpour, and Kevin Smith · 2018
Later among the works it cites.
Neural network ensembles and variational inference revisited
Marcin B Tomczak, Siddharth Swaroop, and Richard E Turner · 2018
Later among the works it cites.
Adversarial distillation of bayesian neural network posteriors
Kuan-Chieh Wang, Paul Vicol, James Lucas, Li Gu, Roger Grosse, and Richard Zemel · 2018
Later among the works it cites.
Deterministic variational inference for robust bayesian neural networks
Anqi Wu, Sebastian Nowozin, Edward Meeds, Richard E Turner, José Miguel Hernández-Lobato, and Alexander L Gaunt · 2018
Later among the works it cites.
Uncertainty estimation via stochastic batch normalization
Andrei Atanov, Arsenii Ashukha, Dmitry Molchanov, Kirill Neklyudov, and Dmitry Vetrov · 2019
Later among the works it cites.
Accelerating monte carlo bayesian inference via approximating predictive uncertainty over simplex
Original
Yufei Cui, Wuguannan Yao, Qiao Li, Antoni B Chan, and Chun Jason Xue · 2019
Later among the works it cites.
Evaluation of neural network uncertainty estimation with application to resource-constrained platforms
Original
Yukun Ding, Jinglan Liu, Jinjun Xiong, and Yiyu Shi · 2019
Later among the works it cites.
Evaluating scalable bayesian deep learning methods for robust computer vision
Original
Fredrik K Gustafsson, Martin Danelljan, and Thomas B Schön · 2019
Later among the works it cites.
Verified uncertainty calibration
Ananya Kumar, Percy S Liang, and Tengyu Ma · 2019
Later among the works it cites.
A simple baseline for bayesian uncertainty in deep learning
Original
Wesley Maddox, Timur Garipov, Pavel Izmailov, Dmitry Vetrov, and Andrew Gordon Wilson · 2019
Later among the works it cites.
Measuring calibration in deep learning
Jeremy Nixon, Mike Dusenberry, Linchuan Zhang, Ghassen Jerfel, and Dustin Tran · 2019
Later among the works it cites.
Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift
Original
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D Sculley, Sebastian Nowozin, Joshua V Dillon, Balaji Lakshminarayanan, and Jasper Snoek · 2019
Later among the works it cites.
Super-convergence: Very fast training of neural networks using large learning rates
Leslie N Smith and Nicholay Topin · 2019
Later among the works it cites.
Evaluating model calibration in classification
Original
Juozas Vaicenavicius, David Widmann, Carl Andersson, Fredrik Lindsten, Jacob Roll, and Thomas B Schon · 2019
Later among the works it cites.
Cyclical stochastic gradient mcmc for bayesian deep learning
Original
Ruqi Zhang, Chunyuan Li, Jianyi Zhang, Changyou Chen, and Andrew Gordon Wilson · 2019
Later among the works it cites.
Hydra: Preserving ensemble diversity for model distillation
Original
Linh Tran, Bastiaan S Veeling, Kevin Roth, Jakub Swiatkowski, Joshua V Dillon, Jasper Snoek, Stephan Mandt, Tim Salimans, Sebastian Nowozin, and Rodolphe Jenatton · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…