Efficient Sparse Attention needs Adaptive Token Release

Understand

In recent years, Large Language Models (LLMs) have demonstrated remarkable capabilities across a wide array of text-centric tasks.

• However, their `large' scale introduces significant computational and storage challenges, particularly in managing the key-value states of the transformer, which limits their wider applicability.
• Therefore, we propose to adaptively release resources from caches and rebuild the necessary key-value states.
• Particularly, we accomplish this by a lightweight controller module to approximate an ideal top-$K$ sparse attention.