2020

Uncertainty Estimation Using a Single Deep Deterministic Neural Network

van Amersfoort, Joost, Smith, Lewis, Teh, Yee Whye et al.

Understand

We propose a method for training a deterministic deep model that can find and reject out of distribution data points at test time with a single forward pass.

Our approach, deterministic uncertainty quantification (DUQ), builds upon ideas of RBF networks.
We scale training in these with a novel loss function and centroid updating scheme and match the accuracy of softmax models.
By enforcing detectability of changes in the input using a gradient penalty, we are able to reliably detect out of distribution data.

Built on

Hybrid models with deep and invertible features
Original
Nalisnick, E., Matsukawa, A., Teh, Y. W., Gorur, D., and Lakshminarayanan, B · 1902
Earlier work this paper cites.
A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines
Hutchinson, M. F · 1990
Earlier work this paper cites.
Improving generalization performance using double backpropagation
Drucker, H. and Le Cun, Y · 1992
Earlier work this paper cites.
Bayesian methods for adaptive models
MacKay, D. J · 1992
Earlier work this paper cites.
The mnist database of handwritten digits, 1998
LeCun, Y., Cortes, C., and Burges, C. J · 1998
Earlier work this paper cites.
Support vector method for novelty detection
Schölkopf, B., Williamson, R. C., Smola, A. J., Shawe-Taylor, J., and Platt, J. C · 2000
Earlier work this paper cites.
notmnist dataset
Bulatov, Y · 2011
Earlier work this paper cites.
Bayesian active learning for classification and preference learning
Original
Houlsby, N., Huszár, F., Ghahramani, Z., and Lengyel, M · 2011
Earlier work this paper cites.
Scikit-learn: Machine learning in python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., et al · 2011
Earlier work this paper cites.
Bayesian learning for neural networks , volume 118
Neal, R. M · 2012
Earlier work this paper cites.
Nice: Non-linear independent components estimation
Original
Dinh, L., Krueger, D., and Bengio, Y · 2014
Earlier work this paper cites.
The cifar-10 dataset
Krizhevsky, A., Nair, V., and Hinton, G · 2014
Earlier work this paper cites.
A least-squares approach to anomaly detection in static and sequential data
Quinn, J. A. and Sugiyama, M · 2014
Earlier work this paper cites.

Similar

Weight uncertainty in neural network
Blundell, C., Cornebise, J., Kavukcuoglu, K., and Wierstra, D · 2015
Cited alongside, same era.
Delving deep into rectifiers: Surpassing human-level performance on imagenet classification
He, K., Zhang, X., Ren, S., and Sun, J · 2015
Cited alongside, same era.
Uncertainty in deep learning
Gal, Y · 2016
Cited alongside, same era.
Dropout as a bayesian approximation: Representing model uncertainty in deep learning
Gal, Y. and Ghahramani, Z · 2016
Cited alongside, same era.
Deep residual learning for image recognition
He, K., Zhang, X., Ren, S., and Sun, J · 2016
Cited alongside, same era.
Deep exploration via bootstrapped dqn
Osband, I., Blundell, C., Pritzel, A., and Van Roy, B · 2016
Cited alongside, same era.
Adversarial examples, uncertainty, and transfer testing robustness in gaussian process hybrid deep networks
Original
Bradshaw, J., Matthews, A. G. d. G., and Ghahramani, Z · 2017
Cited alongside, same era.
Improved training of wasserstein gans
Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., and Courville, A. C · 2017
Cited alongside, same era.
Simple and scalable predictive uncertainty estimation using deep ensembles
Lakshminarayanan, B., Pritzel, A., and Blundell, C · 2017
Cited alongside, same era.
Automatic differentiation in pytorch
Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., and Lerer, A · 2017
Cited alongside, same era.
On the regularization of wasserstein gans
Original
Petzka, H., Fischer, A., and Lukovnicov, D · 2017
Cited alongside, same era.
Pixelcnn++: Improving the pixelcnn with discretized logistic mixture likelihood and other modifications
Original
Salimans, T., Karpathy, A., Chen, X., and Kingma, D. P · 2017
Cited alongside, same era.

Then

Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms
Original
Xiao, H., Rasul, K., and Vollgraf, R · 2017
Later among the works it cites.
i-revnet: Deep invertible networks
Jacobsen, J.-H., Smeulders, A., and Oyallon, E · 2018
Later among the works it cites.
A simple unified framework for detecting out-of-distribution samples and adversarial attacks
Lee, K., Lee, K., Lee, H., and Shin, J · 2018
Later among the works it cites.
Improving the adversarial robustness and interpretability of deep neural networks by regularizing their input gradients
Ross, A. S. and Doshi-Velez, F · 2018
Later among the works it cites.
Understanding measures of uncertainty for adversarial example detection
Original
Smith, L. and Gal, Y · 2018
Later among the works it cites.
A systematic comparison of bayesian deep learning robustness in diabetic retinopathy tasks
Original
Filos, A., Farquhar, S., Gomez, A. N., Rudner, T. G., Kenton, Z., Smith, L., Alizadeh, M., de Kroon, A., and Gal, Y · 2019
Later among the works it cites.
Why relu networks yield high-confidence predictions far away from the training data and how to mitigate the problem
Hein, M., Andriushchenko, M., and Bitterwolf, J · 2019
Later among the works it cites.
Robust learning with jacobian regularization
Original
Hoffman, J., Roberts, D. A., and Yaida, S · 2019
Later among the works it cites.
Connections between support vector machines, wasserstein distance and gradient-penalty gans
Original
Jolicoeur-Martineau, A. and Mitliagkas, I · 2019
Later among the works it cites.
The street view house numbers (svhn) dataset, 2019
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., and Ng, A · 2019
Later among the works it cites.
Likelihood ratios for out-of-distribution detection
Ren, J., Liu, P. J., Fertig, E., Snoek, J., Poplin, R., Depristo, M., Dillon, J., and Lakshminarayanan, B · 2019
Later among the works it cites.
Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift
Snoek, J., Ovadia, Y., Fertig, E., Lakshminarayanan, B., Nowozin, S., Sculley, D., Dillon, J., Ren, J., and Nado, Z · 2019
Later among the works it cites.
Radial Bayesian neural networks: Beyond discrete support in large-scale bayesian deep learning
Farquhar, S., Osborne, M., and Gal, Y · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…