2018

Tunability: Importance of Hyperparameters of Machine Learning Algorithms

Probst, Philipp, Bischl, Bernd, Boulesteix, Anne-Laure

Understand

Modern supervised machine learning algorithms involve hyperparameters that have to be set before running them.

Options for setting hyperparameters are default values from the software package, manual configuration by the user or configuring them for optimal predictive performance by a tuning procedure.
The goal of this paper is two-fold.
Firstly, we formalize the problem of tuning from a statistical point of view, define data-based defaults and suggest general measures quantifying the tunability of hyperparameters of algorithms.

Built on

F-Race and iterated F-Race: An overview
M. Birattari, Z. Yuan, P. Balaprakash, and T. Stützle · 2010
Earlier work this paper cites.
Model selection: Beyond the bayesian/frequentist divide
I. Guyon, A. Saffari, G. Dror, and G. Cawley · 2010
Earlier work this paper cites.
Parameter tuning for configuring and analyzing evolutionary algorithms
A. E. Eiben and S. K. Smit · 2011
Earlier work this paper cites.
Sequential model-based optimization for general algorithm configuration
F. Hutter, H. H. Hoos, and K. Leyton-Brown · 2011
Earlier work this paper cites.
Optimized application of penalized regression methods to diverse genomic data
Original
L. Waldron, M. Pintilie, M.-S. Tsao, F. A. Shepherd, C. Huttenhower, and I. Jurisica · 2011
Earlier work this paper cites.
Random search for hyper-parameter optimization
J. Bergstra and Y. Bengio · 2012
Earlier work this paper cites.
Resampling methods for meta-model validation with recommendations for evolutionary computation
B. Bischl, O. Mersmann, H. Trautmann, and C. Weihs · 2012
Earlier work this paper cites.
Practical bayesian optimization of machine learning algorithms
J. Snoek, H. Larochelle, and R. P. Adams · 2012
Earlier work this paper cites.
Identifying key algorithm parameters and instance features using forward selection
F. Hutter, H. H. Hoos, and K. Leyton-Brown · 2013
Earlier work this paper cites.
OpenML: Networked science in machine learning
J. Vanschoren, J. N. van Rijn, B. Bischl, and L. Torgo · 2013
Earlier work this paper cites.

Similar

Surrogate benchmarks for hyperparameter optimization
K. Eggensperger, F. Hutter, H. H. Hoos, and K. Leyton-Brown · 2014
Cited alongside, same era.
An efficient approach for assessing hyperparameter importance
F. Hutter, H. Hoos, and K. Leyton-Brown · 2014
Cited alongside, same era.
Learning feature-parameter mappings for parameter tuning via the profile expected improvement
J. Bossek, B. Bischl, T. Wagner, and G. Rudolph · 2015
Cited alongside, same era.
mlr: Machine learning in R
B. Bischl, M. Lang, L. Kotthoff, J. Schiffner, J. Richter, E. Studerus, G. Casalicchio, and Z. M. Jones · 2016
Cited alongside, same era.
Analysing differences between algorithm configurations through ablation
C. Fawcett and H. H. Hoos · 2016
Cited alongside, same era.
Cubist: Rule- and instance-based regression modeling , 2016
M. Kuhn, S. Weston, C. Keefer, and N. Coulter · 2016
Cited alongside, same era.
A review of automatic selection methods for machine learning algorithms and hyper-parameter values
G. Luo · 2016
Cited alongside, same era.
Hyper-parameter tuning of a decision tree induction algorithm
R. G. Mantovani, T. Horváth, R. Cerri, A. Carvalho, and J. Vanschoren · 2016
Cited alongside, same era.
Efficient parameter importance analysis via ablation with surrogates
A. Biedenkapp, M. T. Lindauer, K. Eggensperger, F. Hutter, C. Fawcett, and H. H. Hoos · 2017
Cited alongside, same era.

Then

OpenML benchmarking suites and the OpenML100
Original
B. Bischl, G. Casalicchio, M. Feurer, F. Hutter, M. Lang, R. G. Mantovani, J. N. van Rijn, and J. Vanschoren · 2017
Later among the works it cites.
mlrMBO: A modular framework for model-based optimization of expensive black-box functions
Original
B. Bischl, J. Richter, J. Bossek, D. Horn, J. Thomas, and M. Lang · 2017
Later among the works it cites.
OpenML: An R package to connect to the machine learning platform OpenML
G. Casalicchio, J. Bossek, M. Lang, D. Kirchhoff, P. Kerschke, B. Hofner, H. Seibold, J. Vanschoren, and B. Bischl · 2017
Later among the works it cites.
batchtools: Tools for R to work on batch systems
M. Lang, B. Bischl, and D. Surmann · 2017
Later among the works it cites.
Hyperparameter importance across datasets
Original
J. N. van Rijn and F. Hutter · 2017
Later among the works it cites.
Efficient benchmarking of algorithm configurators via model-based surrogates
K. Eggensperger, M. Lindauer, H. H. Hoos, F. Hutter, and K. Leyton-Brown · 2018
Closest in time.
Scalable meta-learning for bayesian optimization
Original
M. Feurer, B. Letham, and E. Bakshy · 2018
Closest in time.
Automatic Exploration of Machine Learning Experiments on OpenML
Original
D. Kühn, P. Probst, J. Thomas, and B. Bischl · 2018
Closest in time.
OpenML R bot benchmark data (final subset), 2018
D. Kühn, P. Probst, J. Thomas, and B. Bischl · 2018
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…