Stochastic Marginal Likelihood Gradients using Neural Tangent Kernels

Understand

Selecting hyperparameters in deep learning greatly impacts its effectiveness but requires manual effort and expertise.

• Recent works show that Bayesian model selection with Laplace approximations can allow to optimize such hyperparameters just like standard neural network parameters using gradients and on the training data.
• However, estimating a single hyperparameter gradient requires a pass through the entire dataset, limiting the scalability of such algorithms.
• In this work, we overcome this issue by introducing lower bounds to the linearized Laplace approximation of the marginal likelihood.