SWAT: Scalable and Efficient Window Attention-based Transformers Acceleration on FPGAs

Understand

Efficiently supporting long context length is crucial for Transformer models.

• The quadratic complexity of the self-attention computation plagues traditional Transformers.
• Sliding window-based static sparse attention mitigates the problem by limiting the attention scope of the input tokens, reducing the theoretical complexity from quadratic to linear.
• Although the sparsity induced by window attention is highly structured, it does not align perfectly with the microarchitecture of the conventional accelerators, leading to suboptimal implementation.