2019

Bayesian Optimization using Deep Gaussian Processes

Hebbal, Ali, Brevault, Loic, Balesdent, Mathieu et al.

Understand

Bayesian Optimization using Gaussian Processes is a popular approach to deal with the optimization of expensive black-box functions.

However, because of the a priori on the stationarity of the covariance matrix of classic Gaussian Processes, this method may not be adapted for non-stationary functions involved in the optimization problem.
To overcome this issue, a new Bayesian Optimization approach is proposed.
It is based on Deep Gaussian Processes as surrogate models instead of classic Gaussian Processes.

Built on

Kriging and automated variogram modeling within a moving window
Timothy C Haas · 1990
Earlier work this paper cites.
Probability, random variables and stochastic processes, 1991
Athanasios Papoulis and Pillai Unnikrishna · 1991
Earlier work this paper cites.
A general regression neural network
Donald F Specht · 1991
Earlier work this paper cites.
Nonparametric estimation of nonstationary spatial covariance structure
Paul D Sampson and Peter Guttorp · 1992
Earlier work this paper cites.
Non stationarity of phenomena related to drought
Ian Cordery and S L. YAO · 1993
Earlier work this paper cites.
Global optimization with nonparametric function fitting
Matthias Schonlau, William J Welch, and D Jones · 1996
Earlier work this paper cites.
Sdo: A statistical method for global optimization
Dennis D Cox and Susan John · 1997
Earlier work this paper cites.
Efficient global optimization of expensive black-box functions
Donald R Jones, Matthias Schonlau, and William J Welch · 1998
Earlier work this paper cites.
Bayesian Gaussian processes for regression and classification
Mark N Gibbs · 1998
Earlier work this paper cites.
Non-stationary spatial modeling
Dave Higdon, Jenise Swall, and John Kern · 1999
Earlier work this paper cites.
A surrogate-model-based method for constrained optimization
Charles Audet, J Denni, Douglas Moore, Andrew Booker, and Paul Frank · 2000
Earlier work this paper cites.
The use of surrogate modeling algorithms to exploit disparities in function computation time within simulation-based optimization
Michael J Sasena, Panos Y Papalambros, and Pierre Goovaerts · 2001
Earlier work this paper cites.
Infinite mixtures of gaussian process experts
Carl E Rasmussen and Zoubin Ghahramani · 2002
Earlier work this paper cites.
Flexibility and efficiency enhancements for constrained global design optimization with kriging approximations
Michael J Sasena · 2002
Earlier work this paper cites.
Gaussian processes for machine learning
Carl Rasmussen and Christopher KI Williams · 2006
Earlier work this paper cites.
Fitting Models of Nonstationary Time Series: An Application to EEG Data
Sreenivas Konda · 2006
Earlier work this paper cites.
Spatial modelling using a new class of nonstationary covariance functions
Christopher J Paciorek and Mark J Schervish · 2006
Earlier work this paper cites.

Similar

Sparse gaussian processes using pseudo-inputs
Edward Snelson and Zoubin Ghahramani · 2006
Cited alongside, same era.
Review of metamodeling techniques in support of engineering design optimization
Gary Wang and Songqing Shan · 2007
Cited alongside, same era.
Non-stationary variogram models for geostatistical sampling optimisation: An empirical investigation using elevation data
Peter M Atkinson and Christopher D Lloyd · 2007
Cited alongside, same era.
A non-stationary covariance-based kriging method for metamodelling in engineering design
Ying Xiong, Wei Chen, Daniel Apley, and Xuru Ding · 2007
Cited alongside, same era.
Stationarity is dead: Whither water management?
Paul CD Milly, Julio Betancourt, Malin Falkenmark, Robert M Hirsch, Zbigniew W Kundzewicz, Dennis P Lettenmaier, and Ronald J Stouffer · 2008
Cited alongside, same era.
Bayesian treed gaussian process models with an application to computer modeling
Robert B Gramacy and Herbert K H Lee · 2008
Cited alongside, same era.
Statistical modeling by wavelets
Brani Vidakovic · 2009
Cited alongside, same era.
Variational learning of inducing variables in sparse gaussian processes
Michalis Titsias · 2009
Cited alongside, same era.
Differential evolution algorithm with strategy adaptation for global numerical optimization
A Kai Qin, Vicky Ling Huang, and Ponnuthurai N Suganthan · 2009
Cited alongside, same era.
Bayesian gaussian process latent variable model
Michalis Titsias and Neil D Lawrence · 2010
Cited alongside, same era.
Portfolio allocation for bayesian optimization
Matthew D Hoffman, Eric Brochu, and Nando de Freitas · 2011
Cited alongside, same era.
Learning non-stationary space-time models for environmental monitoring
Sahil Garg, Amarjeet Singh, and Fabio Ramos · 2012
Cited alongside, same era.

Then

Predictive entropy search for efficient global optimization of black-box functions
José Miguel Hernández-Lobato, Matthew W Hoffman, and Zoubin Ghahramani · 2014
Later among the works it cites.
An entropy search portfolio for bayesian optimization
Original
Bobak Shahriari, Ziyu Wang, Matthew W Hoffman, Alexandre Bouchard-Côté, and Nando de Freitas · 2014
Later among the works it cites.
Adam: A method for stochastic optimization
Original
Diederik P Kingma and Jimmy Ba · 2014
Later among the works it cites.
Deep learning
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton · 2015
Later among the works it cites.
Variational auto-encoded deep gaussian processes
Original
Zhenwen Dai, Andreas Damianou, Javier González, and Neil Lawrence · 2015
Later among the works it cites.
Deep gaussian processes and variational propagation of uncertainty
Andreas Damianou · 2015
Later among the works it cites.
Deep gaussian processes for regression using approximate expectation propagation
Thang Bui, Daniel Hernández-Lobato, Jose Hernandez-Lobato, Yingzhen Li, and Richard Turner · 2016
Later among the works it cites.
Classification and regression trees
Leo Breiman · 2017
Later among the works it cites.
Analysis of thompson sampling for gaussian process optimization in the bandit setting
Original
Kinjal Basu and Souvik Ghosh · 2017
Later among the works it cites.
Doubly stochastic variational inference for deep gaussian processes
Hugh Salimbeni and Marc Deisenroth · 2017
Later among the works it cites.
GPflow: A Gaussian process library using TensorFlow
Alexander G. de G. Matthews, Mark van der Wilk, Tom Nickson, Keisuke. Fujii, Alexis Boukouvalas, Pablo León-Villagrá, Zoubin Ghahramani, and James Hensman · 2017
Later among the works it cites.
A tutorial on bayesian optimization
Original
Peter I Frazier · 2018
Later among the works it cites.
Warped gaussian processes and derivative-based sequential designs for functions with heterogeneous variations
Sébastien Marmin, David Ginsbourger, Jean Baccou, and Jacques Liandrat · 2018
Later among the works it cites.
Inference in deep gaussian processes using stochastic gradient hamiltonian monte carlo
Marton Havasi, José Miguel Hernández-Lobato, and Juan José Murillo-Fuentes · 2018
Later among the works it cites.
Efficient global optimization using deep gaussian processes
Ali Hebbal, Loïc Brevault, Mathieu Balesdent, Ei-Ghazali Taibi, and Nouredine Melab · 2018
Later among the works it cites.
Natural gradients in practice: Non-conjugate variational inference in gaussian process models
Hugh Salimbeni, Stefanos Eleftheriadis, and James Hensman · 2018
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…