A Geometric View of Data Complexity: Efficient Local Intrinsic Dimension Estimation with Diffusion Models

Understand

High-dimensional data commonly lies on low-dimensional submanifolds, and estimating the local intrinsic dimension (LID) of a datum -- i.e.

• the dimension of the submanifold it belongs to -- is a longstanding problem.
• LID can be understood as the number of local factors of variation: the more factors of variation a datum has, the more complex it tends to be.
• Estimating this quantity has proven useful in contexts ranging from generalization in neural networks to detection of out-of-distribution data, adversarial examples, and AI-generated text.