Understanding High-Dimensional Bayesian Optimization

Understand

Recent work reported that simple Bayesian optimization (BO) methods perform well for high-dimensional real-world tasks, seemingly contradicting prior work and tribal knowledge.

• This paper investigates why.
• We identify underlying challenges that arise in high-dimensional BO and explain why recent methods succeed.
• Our empirical analysis shows that vanishing gradients caused by Gaussian process (GP) initialization schemes play a major role in the failures of high-dimensional Bayesian optimization (HDBO) and that methods that promote local search behaviors are better suited for the task.