InfoNCE Loss Provably Learns Cluster-Preserving Representations

Understand

The goal of contrasting learning is to learn a representation that preserves underlying clusters by keeping samples with similar content, e.g.

• the ``dogness'' of a dog, close to each other in the space generated by the representation.
• A common and successful approach for tackling this unsupervised learning problem is minimizing the InfoNCE loss associated with the training samples, where each sample is associated with their augmentations (positive samples such as rotation, crop) and a batch of negative samples (unrelated samples).
• To the best of our knowledge, it was unanswered if the representation learned by minimizing the InfoNCE loss preserves the underlying data clusters, as it only promotes learning a representation that is faithful to augmentations, i.e., an image and its augmentations have the same representation.