2018

Predictive Uncertainty Estimation via Prior Networks

Malinin, Andrey, Gales, Mark

Understand

Estimating how uncertain an AI system is in its predictions is important to improve the safety of such systems.

Uncertainty in predictive can result from uncertainty in model parameters, irreducible data uncertainty and uncertainty due to distributional mismatch between the test and training data distributions.
Different actions might be taken depending on the source of the uncertainty so it is important to be able to distinguish between them.
Recently, baseline tasks and metrics have been defined and several practical methods to estimate uncertainty developed.

Built on

“A practical bayesian framework for backpropagation networks,”
David JC MacKay, · 1992
Earlier work this paper cites.
Bayesian methods for adaptive models
David JC MacKay, · 1992
Earlier work this paper cites.
“Keeping the neural networks simple by minimizing the description length of the weights,”
Geoffrey E. Hinton and Drew van Camp, · 1993
Earlier work this paper cites.
Bayesian learning for neural networks
Radford M. Neal, · 1996
Earlier work this paper cites.
“Gradient-based learning applied to document recognition,”
Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, · 1998
Earlier work this paper cites.
“Metanet: The theory of independent judges,”
M Buscema, · 1998
Earlier work this paper cites.
“Latent Dirichlet Allocation,”
David M. Blei, Andrew Y. Ng, and Michael I. Jordan, · 2003
Earlier work this paper cites.
Dataset Shift in Machine Learning
Joaquin Quiñonero-Candela, · 2009
Earlier work this paper cites.
“Learning multiple layers of features from tiny images,”
Alex Krizhevsky, · 2009
Earlier work this paper cites.
“Recurrent Neural Network Based Language Model,”
Tomas Mikolov, Martin Karafiát, Lukás Burget, Jan Cernocký, and Sanjeev Khudanpur, · 2010
Earlier work this paper cites.
“Practical variational inference for neural networks,”
Alex Graves, · 2011
Earlier work this paper cites.
“Bayesian Learning via Stochastic Gradient Langevin Dynamics,”
Max Welling and Yee Whye Teh, · 2011
Earlier work this paper cites.
“Deep neural networks for acoustic modeling in speech recognition,”
Geoffrey Hinton, Li Deng, Dong Yu, George Dahl, Abdel rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara Sainath, and Brian Kingsbury, · 2012
Earlier work this paper cites.
Machine Learning
Kevin P. Murphy, · 2012
Earlier work this paper cites.
“Linguistic Regularities in Continuous Space Word Representations,”
Tomas Mikolov et al., · 2013
Earlier work this paper cites.

Similar

“Efficient Estimation of Word Representations in Vector Space,” 2013,
Original
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean, · 2013
Cited alongside, same era.
“Multi-digit number recognition from street view imagery using deep convolutional neural networks,” 2013,
Original
Ian J. Goodfellow, Yaroslav Bulatov, Julian Ibarz, Sacha Arnoud, and Vinay D. Shet, · 2013
Cited alongside, same era.
“Deep speech: Scaling up end-to-end speech recognition,” 2014,
Original
Awni Y. Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, and Andrew Y. Ng, · 2014
Cited alongside, same era.
“Fast R-CNN,”
Ross Girshick, · 2015
Cited alongside, same era.
“Very Deep Convolutional Networks for Large-Scale Image Recognition,”
Karen Simonyan and Andrew Zisserman, · 2015
Cited alongside, same era.
“Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission,”
Rich Caruana, Yin Lou, Johannes Gehrke, Paul Koch, Marc Sturm, and Noemie Elhadad, · 2015
Cited alongside, same era.
“Predicting the sequence specificities of DNA- and RNA-binding proteins by deep learning,”
Babak Alipanahi, Andrew Delong, Matthew T. Weirauch, and Brendan J. Frey, · 2015
Cited alongside, same era.
“Weight Uncertainty in Neural Networks,”
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra, · 2015
Cited alongside, same era.
“Variational dropout and the local reparameterization trick,”
Diederik P Kingma, Tim Salimans, and Max Welling, · 2015
Cited alongside, same era.
“Distilling the knowledge in a neural network,” 2015,
Original
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean, · 2015
Cited alongside, same era.
“Human-level concept learning through probabilistic program induction,”
Brenden M. Lake, Ruslan Salakhutdinov, and Joshua B. Tenenbaum, · 2015
Cited alongside, same era.
“LSUN: construction of a large-scale image dataset using deep learning with humans in the loop,” 2015,
Original
Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao, · 2015
Cited alongside, same era.

Then

“A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks,” http://arxiv.org/abs/1610.02136 , 2016,
Original
Dan Hendrycks and Kevin Gimpel, · 2016
Later among the works it cites.
“Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning,”
Yarin Gal and Zoubin Ghahramani, · 2016
Later among the works it cites.
Uncertainty in Deep Learning
Yarin Gal, · 2016
Later among the works it cites.
“Structured and efficient variational deep learning with matrix gaussian posteriors,”
Christos Louizos and Max Welling, · 2016
Later among the works it cites.
“Incorporating Nesterov Momentum into Adam,”
Timothy Dozat, · 2016
Later among the works it cites.
“Learning to Generate Long-term Future via Hierarchical Prediction,”
Ruben Villegas, Jimei Yang, Yuliang Zou, Sungryull Sohn, Xunyu Lin, and Honglak Lee, · 2017
Later among the works it cites.
“Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles,”
B. Lakshminarayanan, A. Pritzel, and C. Blundell, · 2017
Later among the works it cites.
“Multi-Task Learning Using Uncertainty to Weight Losses for Scene Geometry and Semantics,”
A. Kendall, Y. Gal, and R. Cipolla, · 2017
Later among the works it cites.
“What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision,”
A. Kendall and Y. Gal, · 2017
Later among the works it cites.
“Incorporating Uncertainty into Deep Learning for Spoken Language Assessment,”
A. Malinin, A. Ragni, M.J.F. Gales, and K.M. Knill, · 2017
Later among the works it cites.
“Decomposition of uncertainty for active learning and reliable reinforcement learning in stochastic systems,”
Original
Stefan Depeweg, José Miguel Hernández-Lobato, Finale Doshi-Velez, and Steffen Udluft, · 2017
Later among the works it cites.
“Tiny ImageNet,” https://tiny-imagenet.herokuapp.com/ , 2017
Stanford CS231N, · 2017
Later among the works it cites.
“Training confidence-calibrated classifiers for detecting out-of-distribution samples,”
Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin, · 2018
Closest in time.
“Enhancing the reliability of out-of-distribution image detection in neural networks,”
Shiyu Liang, Yixuan Li, and R. Srikant, · 2018
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…