Scalable One-Pass Optimisation of High-Dimensional Weight-Update Hyperparameters by Implicit Differentiation

Understand

Machine learning training methods depend plentifully and intricately on hyperparameters, motivating automated strategies for their optimisation.

• Many existing algorithms restart training for each new hyperparameter choice, at considerable computational cost.
• Some hypergradient-based one-pass methods exist, but these either cannot be applied to arbitrary optimiser hyperparameters (such as learning rates and momenta) or take several times longer to train than their base models.
• We extend these existing methods to develop an approximate hypergradient-based hyperparameter optimiser which is applicable to any continuous hyperparameter appearing in a differentiable model weight update, yet requires only one training episode, with no restarts.