Flare: Anomaly Diagnostics for Divergent LLM Training in GPU Clusters of Thousand-Plus Scale

Understand

The rapid proliferation of large language models has driven the need for efficient GPU training clusters.

• However, it is challenging due to the frequent occurrence of training anomalies.
• Since existing diagnostic tools are narrowly tailored to specific issues, there are gaps in their ability to address anomalies spanning the entire training stack.
• In response, we introduce Flare, a diagnostic framework designed for distributed LLM training at scale.