Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer

Understand

Hyperparameter (HP) tuning in deep learning is an expensive process, prohibitively so for neural networks (NNs) with billions of parameters.

• We show that, in the recently discovered Maximal Update Parametrization (muP), many optimal HPs remain stable even as model size changes.
• This leads to a new HP tuning paradigm we call muTransfer: parametrize the target model in muP, tune the HP indirectly on a smaller model, and zero-shot transfer them to the full-sized model, i.e., without directly tuning the latter at all.
• We verify muTransfer on Transformer and ResNet.