Couplformer:Rethinking Vision Transformer with Coupling Attention Map

Understand

With the development of the self-attention mechanism, the Transformer model has demonstrated its outstanding performance in the computer vision domain.

• However, the massive computation brought from the full attention mechanism became a heavy burden for memory consumption.
• Sequentially, the limitation of memory reduces the possibility of improving the Transformer model.
• To remedy this problem, we propose a novel memory economy attention mechanism named Couplformer, which decouples the attention map into two sub-matrices and generates the alignment scores from spatial information.