2015

Hyperparameter Search in Machine Learning

Claesen, Marc, De Moor, Bart

Understand

We introduce the hyperparameter search problem in the field of machine learning and discuss its main challenges from an optimization perspective.

Machine learning methods attempt to build models that capture some element of interest based on given data.
Most common learning algorithms feature a set of hyperparameters that must be determined before training commences.
The choice of hyperparameters can significantly affect the resulting model's performance, but determining good values can be complex; hence a disciplined, theoretically sound search strategy is essential.

Built on

A leisurely look at the bootstrap, the jackknife, and cross-validation
Bradley Efron and Gail Gong · 1983
Earlier work this paper cites.
Neural networks and the bias/variance dilemma
Stuart Geman, Elie Bienenstock, and René Doursat · 1992
Earlier work this paper cites.
Neural networks for pattern recognition
Christopher M Bishop et al · 1995
Earlier work this paper cites.
A study of cross-validation and bootstrap for accuracy estimation and model selection
Ron Kohavi · 1995
Earlier work this paper cites.
Efficient global optimization of expensive black-box functions
Donald R Jones, Matthias Schonlau, and William J Welch · 1998
Earlier work this paper cites.
Regularization networks and support vector machines
Theodoros Evgeniou, Massimiliano Pontil, and Tomaso Poggio · 2000
Earlier work this paper cites.
Random forests
Leo Breiman · 2001
Earlier work this paper cites.
Best choices for regularization parameters in learning theory: on the bias-variance problem
Felipe Cucker and Steve Smale · 2002
Earlier work this paper cites.
Learning with kernels: support vector machines, regularization, optimization, and beyond
Bernhard Schölkopf and Alexander J Smola · 2002
Earlier work this paper cites.
A practical guide to support vector classification, 2003
Chih-Wei Hsu, Chih-Chung Chang, Chih-Jen Lin, et al · 2003
Earlier work this paper cites.
The entire regularization path for the support vector machine
Trevor Hastie, Saharon Rosset, Robert Tibshirani, and Ji Zhu · 2004
Earlier work this paper cites.

Similar

Optimized particle swarm optimization (OPSO) and its application to artificial neural network training
Michael Meissner, Michael Schmuker, and Gisbert Schneider · 2006
Cited alongside, same era.
Tuning the structure and parameters of a neural network by using hybrid taguchi-genetic algorithm
Jinn-Tsong Tsai, Jyh-Horng Chou, and Tung-Kuan Liu · 2006
Cited alongside, same era.
Support vector machine solvers
Léon Bottou and Chih-Jen Lin · 2007
Cited alongside, same era.
Particle swarm optimization for parameter determination and feature selection of support vector machines
Shih-Wei Lin, Kuo-Ching Ying, Shih-Chieh Chen, and Zne-Jung Lee · 2008
Cited alongside, same era.
ParamILS: an automatic algorithm configuration framework
Frank Hutter, Holger H Hoos, Kevin Leyton-Brown, and Thomas Stützle · 2009
Cited alongside, same era.
F-race and iterated f-race: An overview
Mauro Birattari, Zhi Yuan, Prasanna Balaprakash, and Thomas Stützle · 2010
Cited alongside, same era.
Coupled simulated annealing
Original
Samuel Xavier-de Souza, Johan AK Suykens, Joos Vandewalle, and Désiré Bollé · 2010
Cited alongside, same era.
Algorithms for hyper-parameter optimization
James S Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl · 2011
Cited alongside, same era.
Sequential model-based optimization for general algorithm configuration
Frank Hutter, Holger H Hoos, and Kevin Leyton-Brown · 2011
Cited alongside, same era.
Scikit-learn: Machine learning in Python
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al · 2011
Cited alongside, same era.
Random search for hyper-parameter optimization
James Bergstra and Yoshua Bengio · 2012
Cited alongside, same era.

Then

Large scale distributed deep networks
Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Andrew Senior, Paul Tucker, Ke Yang, Quoc V Le, et al · 2012
Later among the works it cites.
A practical guide to training restricted boltzmann machines
Geoffrey E Hinton · 2012
Later among the works it cites.
Imagenet classification with deep convolutional neural networks
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton · 2012
Later among the works it cites.
Practical Bayesian optimization of machine learning algorithms
Jasper Snoek, Hugo Larochelle, and Ryan P Adams · 2012
Later among the works it cites.
Cross validation and maximum likelihood estimations of hyper-parameters of gaussian processes with model misspecification
François Bachoc · 2013
Later among the works it cites.
Hyperopt: A Python library for optimizing the hyperparameters of machine learning algorithms
James Bergstra, Dan Yamins, and David D Cox · 2013
Later among the works it cites.
Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures
James Bergstra, Daniel Yamins, and David Cox · 2013
Later among the works it cites.
Towards an empirical foundation for assessing Bayesian optimization of hyperparameters
Katharina Eggensperger, Matthias Feurer, Frank Hutter, James Bergstra, Jasper Snoek, Holger Hoos, and Kevin Leyton-Brown · 2013
Later among the works it cites.
Easy hyperparameter search using Optunity
Original
Marc Claesen, Jaak Simm, Dusan Popovic, Yves Moreau, and Bart De Moor · 2014
Later among the works it cites.
EnsembleSVM: A library for ensemble learning using support vector machines
Marc Claesen, Frank De Smet, Johan A.K. Suykens, and Bart De Moor · 2014
Later among the works it cites.
Sequence to sequence learning with neural networks
Ilya Sutskever, Oriol Vinyals, and Quoc VV Le · 2014
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…