Prismatic Synthesis: Gradient-based Data Diversification Boosts Generalization in LLM Reasoning

Understand

Effective generalization in language models depends critically on the diversity of their training data.

• Yet existing diversity metrics often fall short of this goal, relying on surface-level heuristics that are decoupled from model behavior.
• This motivates us to ask: What kind of diversity in training data actually drives generalization in language models -- and how can we measure and amplify it? Through large-scale empirical analyses spanning over 300 training runs, carefully controlled for data scale and quality, we show that data diversity can be a strong predictor of generalization in LLM reasoning -- as measured by average model performance on unseen out-of-distribution benchmarks.
• We introduce G-Vendi, a metric that quantifies diversity via the entropy of model-induced gradients.