LongLLaDA: Unlocking Long Context Capabilities in Diffusion LLMs

Understand

Large Language Diffusion Models, or diffusion LLMs, have emerged as a significant focus in NLP research, with substantial effort directed toward understanding their scalability and downstream task performance.

• However, their long-context capabilities remain unexplored, lacking systematic analysis or methods for context extension.
• In this work, we present the first systematic investigation comparing the long-context performance of diffusion LLMs and traditional auto-regressive LLMs.
• We first identify a unique characteristic of diffusion LLMs, unlike auto-regressive LLMs, they maintain remarkably stable perplexity during direct context extrapolation.