Unveiling Downstream Performance Scaling of LLMs: A Clustering-Based Perspective

Understand

The escalating scale and cost of Large Language Models (LLMs) training necessitate accurate pre-training prediction of downstream task performance for comprehensive understanding of scaling properties.

• This is challenged by: 1) the emergence phenomenon, where unpredictable capabilities appearing suddenly at critical model scales; and 2) uneven task difficulty and inconsistent performance scaling patterns, leading to high metric variability.
• Current prediction methods lack accuracy and reliability.
• We propose a Clustering-On-Difficulty (COD) framework for downstream performance prediction.