Pay Attention to MLPs

Understand

Transformers have become one of the most important architectural innovations in deep learning and have enabled many breakthroughs over the past few years.

• Here we propose a simple network architecture, gMLP, based on MLPs with gating, and show that it can perform as well as Transformers in key language and vision applications.
• Our comparisons show that self-attention is not critical for Vision Transformers, as gMLP can achieve the same accuracy.
• For BERT, our model achieves parity with Transformers on pretraining perplexity and is better on some downstream NLP tasks.