Bayesian Hyperparameter Optimization with BoTorch, GPyTorch and Ax

Understand

Deep learning models are full of hyperparameters, which are set manually before the learning process can start.

• To find the best configuration for these hyperparameters in such a high dimensional space, with time-consuming and expensive model training / validation, is not a trivial challenge.
• Bayesian optimization is a powerful tool for the joint optimization of hyperparameters, efficiently trading off exploration and exploitation of the hyperparameter space.
• In this paper, we discuss Bayesian hyperparameter optimization, including hyperparameter optimization, Bayesian optimization, and Gaussian processes.