SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inference

Understand

An efficient attention implementation is essential for large models due to its quadratic time complexity.

• Fortunately, attention commonly exhibits sparsity, i.e., many values in the attention map are near zero, allowing for the omission of corresponding computations.
• Many studies have utilized the sparse pattern to accelerate attention.
• However, most existing works focus on optimizing attention within specific models by exploiting certain sparse patterns of the attention map.