Wide Attention Is The Way Forward For Transformers?

Understand

The Transformer is an extremely powerful and prominent deep learning architecture.

• In this work, we challenge the commonly held belief in deep learning that going deeper is better, and show an alternative design approach that is building wider attention Transformers.
• We demonstrate that wide single layer Transformer models can compete with or outperform deeper ones in a variety of Natural Language Processing (NLP) tasks when both are trained from scratch.
• The impact of changing the model aspect ratio on Transformers is then studied systematically.