Let the Code LLM Edit Itself When You Edit the Code

Understand

In this work, we investigate a typical scenario in code generation where a developer edits existing code in real time and requests a code assistant, e.g., a large language model, to re-predict the next token or next line on the fly.

• Naively, the LLM needs to re-encode the entire KV cache to provide an accurate prediction.
• However, this process is computationally expensive, especially when the sequence length is long.
• Simply encoding the edited subsequence and integrating it to the original KV cache meets the temporal confusion problem, leading to significantly worse performance.