Transformers without Tears: Improving the Normalization of Self-Attention

Understand

We evaluate three simple, normalization-centric changes to improve Transformer training.

• First, we show that pre-norm residual connections (PreNorm) and smaller initializations enable warmup-free, validation-based training with large learning rates.
• Second, we propose $\ell_2$ normalization with a single scale parameter (ScaleNorm) for faster training and better performance.
• Finally, we reaffirm the effectiveness of normalizing word embeddings to a fixed length (FixNorm).