2020

A Framework for Interdomain and Multioutput Gaussian Processes

van der Wilk, Mark, Dutordoir, Vincent, John, ST et al.

Understand

One obstacle to the use of Gaussian processes (GPs) in large-scale problems, and as a component in deep learning system, is the need for bespoke derivations and implementations for small variations in the model or inference.

In order to improve the utility of GPs we need a modular system that allows rapid implementation and testing, as seen in the neural network community.
We present a mathematical and software framework for scalable approximate inference in GPs, which combines interdomain approximations and multiple outputs.
Our framework, implemented in GPflow, provides a unified interface for many existing multioutput models, as well as more recent convolutional structures.

Built on

“A unifying view of sparse approximate Gaussian process regression”
Joaquin Quiñonero-Candela and Carl Rasmussen · 1959
Earlier work this paper cites.
“Mining geostatistics”
Andre Journel and Charles Huijbregts · 1978
Earlier work this paper cites.
“Doubly Stochastic Variational Bayes for non-Conjugate Inference”
Michalis Titsias and Miguel Lázaro-Gredilla · 1979
Earlier work this paper cites.
“Some Bayesian numerical analysis”
Anthony O’Hagan · 1992
Earlier work this paper cites.
“The open-closed principle”
Robert Martin · 1996
Earlier work this paper cites.
“Expectation propagation for approximate Bayesian inference”
Thomas. Minka · 2001
Earlier work this paper cites.
“Sparse on-line Gaussian processes”
Lehel Csató and Manfred Opper · 2002
Earlier work this paper cites.
“Space and space-time modeling using process convolutions”
Dave Higdon · 2002
Earlier work this paper cites.
“Bayesian Monte Carlo”
Carl Rasmussen and Zoubin Ghahramani · 2003
Earlier work this paper cites.
“Gaussian Process Latent Variable Models for Visualisation of High Dimensional Data”
Neil. Lawrence · 2004
Earlier work this paper cites.
“Dependent Gaussian Processes”
Phillip Boyle and Marcus Frean · 2005
Earlier work this paper cites.
“On learning vector-valued functions”
Charles. Micchelli and Massimiliano Pontil · 2005
Earlier work this paper cites.
“Gaussian processes for machine learning. 2006”
Carl Rasmussen and Christopher.. Williams · 2006
Earlier work this paper cites.
“Sparse Gaussian Processes using Pseudo-inputs”
Edward Snelson and Zoubin Ghahramani · 2006
Earlier work this paper cites.
“Sparse Convolved Gaussian Processes for Multi-output Regression”
Mauricio. Álvarez and Neil. Lawrence · 2009
Earlier work this paper cites.
“Latent Force Models”
Mauricio. Álvarez, David Luengo and Neil. Lawrence · 2009
Earlier work this paper cites.
“Inter-domain Gaussian Processes for Sparse Inference using Inducing Features”
Miguel Lázaro-Gredilla and Aníbal Figueiras-Vidal · 2009
Earlier work this paper cites.

Similar

“Variational Learning of Inducing Variables in Sparse Gaussian Processes”
Michalis Titsias · 2009
Cited alongside, same era.
“Efficient Multioutput Gaussian Processes through Variational Inducing Kernels”
Mauricio. Álvarez, David Luengo, Michalis Titsias and Neil. Lawrence · 2010
Cited alongside, same era.
“Kernels for vector-valued functions: A review”
Mauricio. Álvarez, Lorenzo Rosasco and Neil. Lawrence · 2012
Cited alongside, same era.
“Deep Gaussian Processes”
Andreas. Damianou and Neil. Lawrence · 2013
Cited alongside, same era.
“Gaussian Processes for Big Data”
James Hensman, Nicoló Fusi and Neil. Lawrence · 2013
Cited alongside, same era.
“Stochastic Variational Inference”
Matthew. Hoffman, David. Blei, Chong Wang and John Paisley · 2013
Cited alongside, same era.
“Auto-encoding variational Bayes”
Original
Diederik. Kingma and Max Welling · 2013
Cited alongside, same era.
“Black Box Variational Inference”
Rajesh Ranganath, Sean Gerrish and David. Blei · 2014
Cited alongside, same era.
“Stochastic Backpropagation and Approximate Inference in Deep Generative Models”
Danilo. Rezende, Shakir Mohamed and Daan Wierstra · 2014
Cited alongside, same era.
“Multiple Dispatch” https://multiple-dispatch.readthedocs.io/en/latest/ , 2014
Matthew Rocklin · 2014
Cited alongside, same era.
“MCMC for Variationally Sparse Gaussian Processes”
James Hensman, Alexander. de G. and Zoubin Ghahramani · 2015
Cited alongside, same era.
“Scalable variational Gaussian process classification”
James Hensman, Alexander.. Matthews and Zoubin Ghahramani · 2015
Cited alongside, same era.

Then

“A Unifying Framework for Gaussian Process Pseudo-Point Approximations using Power Expectation Propagation”
Thang. Bui, Josiah Yan and Richard. Turner · 2017
Later among the works it cites.
“Variational Fourier features for Gaussian processes”
James Hensman, Nicolas Durrande and Arno Solin · 2017
Later among the works it cites.
“Scalable Gaussian process inference using variational methods”, 2017
Alexander.. Matthews · 2017
Later among the works it cites.
“GPflow: A Gaussian Process Library using TensorFlow”
Alexander.. Matthews, Mark van Wilk, Tom Nickson, Keisuke Fujii, Alexis Boukouvalas, Pablo León-Villagrá, Zoubin Ghahramani and James Hensman · 2017
Later among the works it cites.
“Doubly Stochastic Variational Inference for Deep Gaussian Processes”
Hugh Salimbeni and Marc Deisenroth · 2017
Later among the works it cites.
“Convolutional Gaussian Processes”
Mark van Wilk, Carl Rasmussen and James Hensman · 2017
Later among the works it cites.
“Deep convolutional Gaussian processes”
Original
Kenneth Blomqvist, Samuel Kaski and Markus Heinonen · 2018
Later among the works it cites.
“Importance Weighting and Variational Inference”
Justin Domke and Daniel Sheldon · 2018
Later among the works it cites.
“Gaussian Process Conditional Density Estimation”
Vincent Dutordoir, Hugh Salimbeni, James Hensman and Marc Deisenroth · 2018
Later among the works it cites.
“Rates of Convergence for Sparse Variational Gaussian Process Regression”
David Burt, Carl Rasmussen and Mark Van · 2019
Later among the works it cites.
“Convolutional Gaussian Processes” https://gpflow.readthedocs.io/en/develop/notebooks/advanced/convolutional.html , 2019
GPflow documentation · 2019
Later among the works it cites.
“Multi-output Gaussian Processes in GPflow” https://gpflow.readthedocs.io/en/develop/notebooks/advanced/multioutput.html , 2019
GPflow documentation · 2019
Later among the works it cites.
“Variational Fourier features in GPflow” https://gpflow.readthedocs.io/en/develop/notebooks/advanced/variational_fourier_features.html , 2019
GPflow documentation · 2019
Later among the works it cites.
“Deep gaussian processes with importance-weighted variational inference”
Hugh Salimbeni, Vincent Dutordoir, James Hensman and Marc Deisenroth · 2019
Later among the works it cites.
“Sparse Gaussian process approximations and applications”, 2019
Mark van Wilk · 2019
Later among the works it cites.
“Bayesian Image Classification with Deep Convolutional Gaussian Processes”
Vincent Dutordoir, Mark van der Wilk, Artem Artemev and James Hensman · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…