DeepNet: Scaling Transformers to 1,000 Layers

Understand

In this paper, we propose a simple yet effective method to stabilize extremely deep Transformers.

• Specifically, we introduce a new normalization function (DeepNorm) to modify the residual connection in Transformer, accompanying with theoretically derived initialization.
• In-depth theoretical analysis shows that model updates can be bounded in a stable way.
• The proposed method combines the best of two worlds, i.e., good performance of Post-LN and stable training of Pre-LN, making DeepNorm a preferred alternative.