EpMAN: Episodic Memory AttentioN for Generalizing to Longer Contexts

Understand

Recent advances in Large Language Models (LLMs) have yielded impressive successes on many language tasks.

• However, efficient processing of long contexts using LLMs remains a significant challenge.
• We introduce \textbf{EpMAN} -- a method for processing long contexts in an \textit{episodic memory} module while \textit{holistically attending to} semantically relevant context chunks.
• The output of \textit{episodic attention} is then used to reweigh the decoder's self-attention to the stored KV cache of the context during training and generation.