DMFormer: Closing the Gap Between CNN and Vision Transformers

Understand

Vision transformers have shown excellent performance in computer vision tasks.

• As the computation cost of their self-attention mechanism is expensive, recent works tried to replace the self-attention mechanism in vision transformers with convolutional operations, which is more efficient with built-in inductive bias.
• However, these efforts either ignore multi-level features or lack dynamic prosperity, leading to sub-optimal performance.
• In this paper, we propose a Dynamic Multi-level Attention mechanism (DMA), which captures different patterns of input images by multiple kernel sizes and enables input-adaptive weights with a gating mechanism.