KVSink: Understanding and Enhancing the Preservation of Attention Sinks in KV Cache Quantization for LLMs

Understand

Key-Value (KV) cache quantization has become a widely adopted optimization technique for efficient large language models (LLMs) inference by reducing KV cache memory usage and mitigating memory-bound constraints.

• Recent studies have emphasized the importance of preserving the original precision of KVs for the first few tokens to ensure the protection of attention sinks.
• While this approach has proven effective in mitigating performance degradation, its underlying principles remain insufficiently understood.
• Moreover, it fails to address the recent discovery that attention sinks can emerge beyond the initial token positions.