Staged Training for Transformer Language Models

Understand

The current standard approach to scaling transformer language models trains each model size from a different random initialization.

• As an alternative, we consider a staged training setup that begins with a small model and incrementally increases the amount of compute used for training by applying a "growth operator" to increase the model depth and width.
• By initializing each stage with the output of the previous one, the training process effectively re-uses the compute from prior stages and becomes more efficient.
• Our growth operators each take as input the entire training state (including model parameters, optimizer state, learning rate schedule, etc.) and output a new training state from which training continues.