A Theoretical Study of Inductive Biases in Contrastive Learning

Understand

Understanding self-supervised learning is important but challenging.

• Previous theoretical works study the role of pretraining losses, and view neural networks as general black boxes.
• However, the recent work of Saunshi et al.
• argues that the model architecture -- a component largely ignored by previous works -- also has significant influences on the downstream performance of self-supervised learning.