Estimating the intrinsic dimension of datasets by a minimal neighborhood information

Understand

Analyzing large volumes of high-dimensional data is an issue of fundamental importance in data science, molecular simulations and beyond.

• Several approaches work on the assumption that the important content of a dataset belongs to a manifold whose Intrinsic Dimension (ID) is much lower than the crude large number of coordinates.
• Such manifold is generally twisted and curved, in addition points on it will be non-uniformly distributed: two factors that make the identification of the ID and its exploitation really hard.
• Here we propose a new ID estimator using only the distance of the first and the second nearest neighbor of each point in the sample.