On the Transformer Growth for Progressive BERT Training

Understand

Due to the excessive cost of large-scale language model pre-training, considerable efforts have been made to train BERT progressively -- start from an inferior but low-cost model and gradually grow the model to increase the computational complexity.

• Our objective is to advance the understanding of Transformer growth and discover principles that guide progressive training.
• First, we find that similar to network architecture search, Transformer growth also favors compound scaling.
• Specifically, while existing methods only conduct network growth in a single dimension, we observe that it is beneficial to use compound growth operators and balance multiple dimensions (e.g., depth, width, and input length of the model).