Towards the Sparseness of Projection Head in Self-Supervised Learning

Understand

In recent years, self-supervised learning (SSL) has emerged as a promising approach for extracting valuable representations from unlabeled data.

• One successful SSL method is contrastive learning, which aims to bring positive examples closer while pushing negative examples apart.
• Many current contrastive learning approaches utilize a parameterized projection head.
• Through a combination of empirical analysis and theoretical investigation, we provide insights into the internal mechanisms of the projection head and its relationship with the phenomenon of dimensional collapse.