Contrastive learning, multi-view redundancy, and linear models

Understand

Self-supervised learning is an empirically successful approach to unsupervised learning based on creating artificial supervised learning problems.

• A popular self-supervised approach to representation learning is contrastive learning, which leverages naturally occurring pairs of similar and dissimilar data points, or multiple views of the same data.
• This work provides a theoretical analysis of contrastive learning in the multi-view setting, where two views of each datum are available.
• The main result is that linear functions of the learned representations are nearly optimal on downstream prediction tasks whenever the two views provide redundant information about the label.