TRANSOM: An Efficient Fault-Tolerant System for Training LLMs

Understand

Large language models (LLMs) with hundreds of billions or trillions of parameters, represented by chatGPT, have achieved profound impact on various fields.

• However, training LLMs with super-large-scale parameters requires large high-performance GPU clusters and long training periods lasting for months.
• Due to the inevitable hardware and software failures in large-scale clusters, maintaining uninterrupted and long-duration training is extremely challenging.
• As a result, A substantial amount of training time is devoted to task checkpoint saving and loading, task rescheduling and restart, and task manual anomaly checks, which greatly harms the overall training efficiency.