Sparse is Enough in Scaling Transformers

Understand

Large Transformer models yield impressive results on many tasks, but are expensive to train, or even fine-tune, and so slow at decoding that their use and study becomes out of reach.

• We address this problem by leveraging sparsity.
• We study sparse variants for all layers in the Transformer and propose Scaling Transformers, a family of next generation Transformer models that use sparse layers to scale efficiently and perform unbatched decoding much faster than the standard Transformer as we scale up the model size.
• Surprisingly, the sparse layers are enough to obtain the same perplexity as the standard Transformer with the same number of parameters.