SnapKV: LLM Knows What You are Looking for Before Generation

Understand

Large Language Models (LLMs) have made remarkable progress in processing extensive contexts, with the Key-Value (KV) cache playing a vital role in enhancing their performance.

• However, the growth of the KV cache in response to increasing input length poses challenges to memory and time efficiency.
• To address this problem, this paper introduces SnapKV, an innovative and fine-tuning-free approach that efficiently minimizes KV cache size while still delivering comparable performance in real-world applications.
• We discover that each attention head in the model consistently focuses on specific prompt attention features during generation.