Early Stopping is Nonparametric Variational Inference

Understand

We show that unconverged stochastic gradient descent can be interpreted as a procedure that samples from a nonparametric variational approximate posterior distribution.

• This distribution is implicitly defined as the transformation of an initial distribution by a sequence of optimization updates.
• By tracking the change in entropy over this sequence of transformations during optimization, we form a scalable, unbiased estimate of the variational lower bound on the log marginal likelihood.
• We can use this bound to optimize hyperparameters instead of using cross-validation.