Losing dimensions: Geometric memorization in generative diffusion

Understand

Diffusion models power leading generative AI, but when and how they memorize training data, especially on low-dimensional manifolds, remains unclear.

• We find memorization emerges gradually, not abruptly: as data become scarce, diffusion models experience a smooth collapse where their capacity to vary across independent directions diminishes.
• Measuring latent dimensionality via the learned score field, we reveal how generative behavior increasingly centers on a few examples while other variations "freeze out".
• We propose a geometric memorization theory, showing that salient features collapse first, then finer details, leading to near point-wise replication.