2020

Beyond the Mean-Field: Structured Deep Gaussian Processes Improve the Predictive Uncertainties

Lindinger, Jakob, Reeb, David, Lippert, Christoph et al.

Understand

Deep Gaussian Processes learn probabilistic data representations for supervised learning by cascading multiple Gaussian Processes.

While this model family promises flexible predictive distributions, exact inference is not tractable.
Approximate inference techniques trade off the ability to closely resemble the posterior distribution against speed of convergence and computational efficiency.
We propose a novel Gaussian variational family that allows for retaining covariances between latent processes while achieving fast convergence by marginalising out all global latent variables.

Built on

Natural gradient works efficiently in learning
Shun-Ichi Amari · 1998
Earlier work this paper cites.
Early stopping-but when?
Lutz Prechelt · 1998
Earlier work this paper cites.
Gaussian processes for machine learning
Carl Rasmussen and Christopher Williams · 2005
Earlier work this paper cites.
Diagonally-striped matrices and approximate inverse preconditioners
Dennis Smolarski · 2006
Earlier work this paper cites.
Sparse gaussian processes using pseudo-inputs
Edward Snelson and Zoubin Ghahramani · 2006
Earlier work this paper cites.
Variational learning of inducing variables in sparse gaussian processes
Michalis Titsias · 2009
Earlier work this paper cites.
Two problems with variational expectation maximisation for time-series models
Richard Turner and Maneesh Sahani · 2011
Earlier work this paper cites.
Practical bayesian optimization of machine learning algorithms
Jasper Snoek, Hugo Larochelle, and Ryan P Adams · 2012
Earlier work this paper cites.
Deep gaussian processes
Andreas Damianou and Neil Lawrence · 2013
Earlier work this paper cites.
Gaussian processes for big data
James Hensman, Nicolo Fusi, and Neil Lawrence · 2013
Earlier work this paper cites.
Avoiding pathologies in very deep networks
David Duvenaud, Oren Rippel, Ryan P. Adams, and Zoubin Ghahramani · 2014
Earlier work this paper cites.
Stochastic backpropagation and approximate inference in deep generative models
Danilo Rezende, Shakir Mohamed, and Daan Wierstra · 2014
Earlier work this paper cites.
Adam: A method for stochastic optimization
Diederik Kingma and Jimmy Ba · 2015
Earlier work this paper cites.

Similar

Variational dropout and the local reparameterization trick
Diederik Kingma, Tim Salimans, and Max Welling · 2015
Cited alongside, same era.
Optimizing neural networks with kronecker-factored approximate curvature
James Martens and Roger Grosse · 2015
Cited alongside, same era.
Tensorflow: a system for large-scale machine learning
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al · 2016
Cited alongside, same era.
Concrete problems in AI safety
Original
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané · 2016
Cited alongside, same era.
Deep gaussian processes for regression using approximate expectation propagation
Thang Bui, Daniel Hernández-Lobato, Jose Hernandez-Lobato, Yingzhen Li, and Richard Turner · 2016
Cited alongside, same era.
Mastering the game of go with deep neural networks and tree search
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al · 2016
Cited alongside, same era.
Variational inference: A review for statisticians
David Blei, Alp Kucukelbir, and Jon McAuliffe · 2017
Cited alongside, same era.
Leveraging uncertainty information from deep neural networks for disease detection
Christian Leibig, Vaneeda Allken, Murat Seçkin Ayhan, Philipp Berens, and Siegfried Wahl · 2017
Cited alongside, same era.
Gpflow: A gaussian process library using tensorflow
Alexander Matthews, Mark Van Der Wilk, Tom Nickson, Keisuke Fujii, Alexis Boukouvalas, Pablo León-Villagrá, Zoubin Ghahramani, and James Hensman · 2017
Cited alongside, same era.
Doubly stochastic variational inference for deep gaussian processes
Hugh Salimbeni and Marc Deisenroth · 2017
Cited alongside, same era.
Inference in deep gaussian processes using stochastic gradient hamiltonian monte carlo
Marton Havasi, José Lobato, and Juan Fuentes · 2018
Cited alongside, same era.
Scalable gaussian processes with billions of inducing inputs via tensor train decomposition
Pavel Izmailov, Alexander Novikov, and Dmitry Kropotov · 2018
Cited alongside, same era.

Then

Bayesian alignments of warped multi-output gaussian processes
Markus Kaiser, Clemens Otte, Thomas Runkler, and Carl Henrik Ek · 2018
Later among the works it cites.
Learning gaussian processes by minimizing pac-bayesian generalization bounds
David Reeb, Andreas Doerr, Sebastian Gerwinn, and Barbara Rakitsch · 2018
Later among the works it cites.
Natural gradients in practice: non-conjugate variational inference in gaussian process models
Hugh Salimbeni, Stefanos Eleftheriadis, and James Hensman · 2018
Later among the works it cites.
Safe active learning for time-series modeling with gaussian processes
Christoph Zimmer, Mona Meister, and Duy Nguyen-Tuong · 2018
Later among the works it cites.
Multi-resolution multi-task gaussian processes
Oliver Hamelijnck, Theodoros Damoulas, Kangrui Wang, and Mark Girolami · 2019
Later among the works it cites.
Bayesian optimization using deep gaussian processes
Original
Ali Hebbal, Loic Brevault, Mathieu Balesdent, El-Ghazali Talbi, and Nouredine Melab · 2019
Later among the works it cites.
Deep learning with differential gaussian process flows
Pashupati Hegde, Markus Heinonen, Harri Lähdesmäki, and Samuel Kaski · 2019
Later among the works it cites.
Deep gaussian processes with importance-weighted variational inference
Hugh Salimbeni, Vincent Dutordoir, James Hensman, and Marc Peter Deisenroth · 2019
Later among the works it cites.
Can you trust your model’s uncertainty? Evaluating predictive uncertainty under dataset shift
Jasper Snoek, Yaniv Ovadia, Emily Fertig, Balaji Lakshminarayanan, Sebastian Nowozin, D Sculley, Joshua Dillon, Jie Ren, and Zachary Nado · 2019
Later among the works it cites.
Exact gaussian processes on a million data points
Ke Wang, Geoff Pleiss, Jacob Gardner, Stephen Tyree, Kilian Q. Weinberger, and Andrew Gordon Wilson · 2019
Later among the works it cites.
Implicit posterior variational inference for deep gaussian processes
Haibin Yu, Yizhou Chen, Zhongxiang Dai, Kian Hsiang Low, and Patrick Jaillet · 2019
Later among the works it cites.
Compositional uncertainty in deep gaussian processes
Ivan Ustyuzhaninov, Ieva Kazlauskaite, Markus Kaiser, Erik Bodin, Neill D. F. Campbell, and Carl Henrik Ek · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…