2021

Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020

Turner, Ryan, Eriksson, David, McCourt, Michael et al.

Understand

This paper presents the results and insights from the black-box optimization (BBO) challenge at NeurIPS 2020 which ran from July-October, 2020.

The challenge emphasized the importance of evaluating derivative-free optimizers for tuning the hyperparameters of machine learning models.
This was the first black-box optimization challenge with a machine learning emphasis.
It was based on tuning (validation set) performance of standard machine learning models on real datasets.

Built on

pySOT and POAP: An event-driven asynchronous framework for surrogate optimization, 2019a
Original
D. Eriksson, D. Bindel, and C. A. Shoemaker · 1908
Earlier work this paper cites.
A simplex method for function minimization
J. A. Nelder and R. Mead · 1965
Earlier work this paper cites.
On the limited memory BFGS method for large scale optimization
D. C. Liu and J. Nocedal · 1989
Earlier work this paper cites.
Efficient global optimization of expensive black-box functions
D. R. Jones, M. Schonlau, and W. J. Welch · 1998
Earlier work this paper cites.
Reducing the time complexity of the derandomized evolution strategy with covariance matrix adaptation (CMA-ES)
N. Hansen, S. D. Müller, and P. Koumoutsakos · 2003
Earlier work this paper cites.
Gaussian processes in machine learning
C. E. Rasmussen · 2003
Earlier work this paper cites.
Scattered Data Approximation , volume 17
H. Wendland · 2004
Earlier work this paper cites.
ParEGO: A hybrid algorithm with on-line landscape approximation for expensive multiobjective optimization problems
J. Knowles · 2006
Earlier work this paper cites.
A stochastic radial basis function method for the global optimization of expensive functions
R. G. Regis and C. A. Shoemaker · 2007
Earlier work this paper cites.
Engineering Design via Surrogate Modelling: A Practical Guide
A. Forrester, A. Sobester, and A. Keane · 2008
Earlier work this paper cites.
The knowledge-gradient policy for correlated normal beliefs
P. Frazier, W. Powell, and S. Dayanik · 2009
Earlier work this paper cites.
Introduction to Evolutionary Algorithms
X. Yu and M. Gen · 2010
Earlier work this paper cites.
Hands-On Pattern Recognition Challenges in Machine Learning , volume 1
I. Guyon, G. Cawley, G. Dror, and A. Saffari · 2011
Earlier work this paper cites.
Portfolio allocation for Bayesian optimization
M. Hoffman, E. Brochu, and N. de Freitas · 2011
Earlier work this paper cites.
The knowledge-gradient algorithm for sequencing experiments in drug discovery
D. M. Negoescu, P. I. Frazier, and W. B. Powell · 2011
Earlier work this paper cites.
Entropy search for information-efficient global optimization
P. Hennig and C. J. Schuler · 2012
Earlier work this paper cites.
Practical Bayesian optimization of machine learning algorithms
J. Snoek, H. Larochelle, and R. P. Adams · 2012
Earlier work this paper cites.
Model based learning of sigma points in unscented Kalman filtering
R. Turner and C. E. Rasmussen · 2012
Earlier work this paper cites.
OpenTuner: An extensible framework for program autotuning
J. Ansel, S. Kamil, K. Veeramachaneni, J. Ragan-Kelley, J. Bosboom, U.-M. O’Reilly, and S. Amarasinghe · 2014
Earlier work this paper cites.
Bayesian optimization with inequality constraints
J. R. Gardner, M. J. Kusner, Z. E. Xu, K. Q. Weinberger, and J. P. Cunningham · 2014
Earlier work this paper cites.
Hyperopt: A Python library for model selection and hyperparameter optimization
J. Bergstra, B. Komer, C. Eliasmith, D. Yamins, and D. D. Cox · 2015
Earlier work this paper cites.
Kernel-based approximation methods using MATLAB , volume 19
G. E. Fasshauer and M. J. McCourt · 2015
Earlier work this paper cites.
Predictive entropy search for Bayesian optimization with unknown constraints
J. M. Hernández-Lobato, M. A. Gelbart, M. W. Hoffman, R. P. Adams, and Z. Ghahramani · 2015
Earlier work this paper cites.

Similar

Bayesian optimization for learning gaits under uncertainty
R. Calandra, A. Seyfarth, J. Peters, and M. P. Deisenroth · 2016
Cited alongside, same era.
Bayesian optimization for materials design
P. I. Frazier and J. Wang · 2016
Cited alongside, same era.
Batch Bayesian optimization via local penalization
J. González, Z. Dai, P. Hennig, and N. D. Lawrence · 2016
Cited alongside, same era.
Modeling an augmented Lagrangian for blackbox constrained optimization
R. B. Gramacy, G. A. Gray, S. Le Digabel, H. K. Lee, P. Ranjan, G. Wells, and S. M. Wild · 2016
Cited alongside, same era.
COCO: A platform for comparing continuous optimizers in a black-box setting
Original
N. Hansen, A. Auger, O. Mersmann, T. Tusar, and D. Brockhoff · 2016
Cited alongside, same era.
GPyOpt: A Bayesian optimization framework in Python, 2016
The GPyOpt authors · 2016
Cited alongside, same era.
COMBO: An efficient Bayesian optimization library for materials science
T. Ueno, T. D. Rhone, Z. Hou, T. Mizoguchi, and K. Tsuda · 2016
Cited alongside, same era.
The parallel knowledge gradient method for batch Bayesian optimization
J. Wu and P. I. Frazier · 2016
Cited alongside, same era.
Parallel and distributed Thompson sampling for large-scale accelerated exploration of chemical space
J. M. Hernández-Lobato, J. Requeima, E. O. Pyzer-Knapp, and A. Aspuru-Guzik · 2017
Cited alongside, same era.
Designing nanostructures for phonon transport via Bayesian optimization
S. Ju, T. Shiga, L. Feng, Z. Hou, K. Tsuda, and J. Shiomi · 2017
Cited alongside, same era.
RoBO: A flexible and robust Bayesian optimization framework in Python
A. Klein, S. Falkner, N. Mansur, and F. Hutter · 2017
Cited alongside, same era.
GPflowOpt: A Bayesian optimization library using TensorFlow
Original
N. Knudde, J. van der Herten, T. Dhaene, and I. Couckuyt · 2017
Cited alongside, same era.

Then

Analysis of the AutoML challenge series 2015–2018
I. Guyon, L. Sun-Hosoya, M. Boullé, H. J. Escalante, S. Escalera, Z. Liu, D. Jajetic, B. Ray, M. Saeed, M. Sebag, A. Statnikov, W. Tu, and E. Viegas · 2019
Later among the works it cites.
Creating glasswing butterfly-inspired durable antifogging superomniphobic supertransmissive, superclear nanostructured glass through Bayesian learning and optimization
S. Haghanifar, M. McCourt, B. Cheng, J. Wuenschell, P. Ohodnicki, and P. W. Leu · 2019
Later among the works it cites.
CMA-ES/pycma on Github
N. Hansen, Y. Akimoto, and P. Baudis · 2019
Later among the works it cites.
Constrained Bayesian optimization with noisy experiments
B. Letham, B. Karrer, G. Ottoni, E. Bakshy, et al · 2019
Later among the works it cites.
Towards automated deep learning: Analysis of the AutoDL challenge series 2019
Z. Liu, Z. Xu, S. Rajaa, M. Madadi, J. C. J. Junior, S. Escalera, A. Pavao, S. Treguer, W.-W. Tu, and I. Guyon · 2019
Later among the works it cites.
Black box optimization competition BBComp, 2019
I. Loshchilov and T. Glasmachers · 2019
Later among the works it cites.
ProBO: A framework for using probabilistic programming in Bayesian optimization
Original
W. Neiswanger, K. Kandasamy, B. Poczos, J. Schneider, and E. Xing · 2019
Later among the works it cites.
Emulation of physical processes with Emukit
A. Paleyes, M. Pullin, M. Mahsereci, N. Lawrence, and J. González · 2019
Later among the works it cites.
The Bayes opt benchmark documentation, 2019
R. Turner · 2019
Later among the works it cites.
Squirrel: A switching hyperparameter optimizer, 2020
N. Awad, G. Shala, D. Deng, N. Mallik, M. Feurer, K. Eggensperger, A. Biedenkapp, D. Vermetten, H. Wang, C. Doerr, M. Lindauer, and F. Hutter · 2020
Later among the works it cites.
Botorch: A framework for efficient Monte-Carlo Bayesian optimization
M. Balandat, B. Karrer, D. R. Jiang, S. Daulton, B. Letham, A. G. Wilson, and E. Bakshy · 2020
Later among the works it cites.
Better call surrogates: A hybrid evolutionary algorithm for hyperparameter optimization, 2020
S. Biswas, A. D. Cobb, A. Sistrunk, N. Ramakrishnan, and B. Jalaian · 2020
Later among the works it cites.
Survey of machine-learning experimental methods at NeurIPS2019 and ICLR2020
X. Bouthillier and G. Varoquaux · 2020
Later among the works it cites.
HEBO: Heteroscedastic evolutionary Bayesian optimisation, 2020
A. I. Cowen-Rivers, W. Lyu, Z. Wang, R. Tutunov, H. Jianye, J. Wang, and H. B. Ammar · 2020
Later among the works it cites.
Differentiable expected hypervolume improvement for parallel multi-objective Bayesian optimization
S. Daulton, M. Balandat, and E. Bakshy · 2020
Later among the works it cites.
Tuning hyperparameters without grad students: Scalable and robust Bayesian optimisation with Dragonfly
K. Kandasamy, K. R. Vysyaraju, W. Neiswanger, B. Paria, C. R. Collins, J. Schneider, B. Poczos, and E. P. Xing · 2020
Later among the works it cites.
Adaptive local Bayesian optimization over multiple discrete variables, 2020
T. Kim, J. Ahn, N. Kim, and S. Yun · 2020
Later among the works it cites.
Solving black-box optimization challenge via learning search space partition for local Bayesian optimization, 2020
M. Sazanovich, A. Nikolskaya, Y. Belousov, and A. Shpilman · 2020
Later among the works it cites.
Team Optuna developers’ method for black-box optimization challenge 2020, 2020
M. Shibata, T. Yanase, H. Imamura, M. Nomura, T. Ohta, S. Sano, and H. V. Yamazaki · 2020
Later among the works it cites.
Learning search space partition for black-box optimization using Monte Carlo tree search
Original
L. Wang, R. Fonseca, and Y. Tian · 2020
Later among the works it cites.
Higher performance for AutoML: The benefit of various ensemble Bayesian optimization strategy, 2020
J. Wu, M. Cao, L. Shan, and Q. Ying · 2020
Later among the works it cites.
Scalable constrained Bayesian optimization
D. Eriksson and M. Poloczek · 2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…