MetaFormer Is Actually What You Need for Vision

Understand

Transformers have shown great potential in computer vision tasks.

• A common belief is their attention-based token mixer module contributes most to their competence.
• However, recent works show the attention-based module in Transformers can be replaced by spatial MLPs and the resulted models still perform quite well.
• Based on this observation, we hypothesize that the general architecture of the Transformers, instead of the specific token mixer module, is more essential to the model's performance.