A Comprehensive Survey on Long Context Language Modeling

Understand

Efficient processing of long contexts has been a persistent pursuit in Natural Language Processing.

• With the growing number of long documents, dialogues, and other textual data, it is important to develop Long Context Language Models (LCLMs) that can process and analyze extensive inputs in an effective and efficient way.
• In this paper, we present a comprehensive survey on recent advances in long-context modeling for large language models.
• Our survey is structured around three key aspects: how to obtain effective and efficient LCLMs, how to train and deploy LCLMs efficiently, and how to evaluate and analyze LCLMs comprehensively.