CompRess: Self-Supervised Learning by Compressing Representations

Understand

Self-supervised learning aims to learn good representations with unlabeled data.

• Recent works have shown that larger models benefit more from self-supervised learning than smaller models.
• As a result, the gap between supervised and self-supervised learning has been greatly reduced for larger models.
• In this work, instead of designing a new pseudo task for self-supervised learning, we develop a model compression method to compress an already learned, deep self-supervised model (teacher) to a smaller one (student).