MEMO: Fine-grained Tensor Management For Ultra-long Context LLM Training

Understand

Nowadays, Large Language Models (LLMs) have been trained using extended context lengths to foster more creative applications.

• However, long context training poses great challenges considering the constraint of GPU memory.
• It not only leads to substantial activation memory consumption during training, but also incurs considerable memory fragmentation.
• To facilitate long context training, existing frameworks have adopted strategies such as recomputation and various forms of parallelisms.