XC-Cache: Cross-Attending to Cached Context for Efficient LLM Inference

Understand

In-context learning (ICL) approaches typically leverage prompting to condition decoder-only language model generation on reference information.

• Just-in-time processing of a context is inefficient due to the quadratic cost of self-attention operations, and caching is desirable.
• However, caching transformer states can easily require almost as much space as the model parameters.
• When the right context isn't known in advance, caching ICL can be challenging.