KVLink: Accelerating Large Language Models via Efficient KV Cache Reuse

Understand

We describe KVLink, an approach for efficient key-value (KV) cache reuse in large language models (LLMs).

• In many LLM applications, different inputs can share overlapping context, such as the same retrieved document appearing in multiple queries.
• However, the LLMs still need to encode the entire context for each query, leading to redundant computation.
• In this paper, we investigate a new strategy to eliminate such inefficiency, where the KV cache of each document is precomputed independently.