Sigmoid Loss for Language Image Pre-Training

Understand

We propose a simple pairwise Sigmoid loss for Language-Image Pre-training (SigLIP).

• Unlike standard contrastive learning with softmax normalization, the sigmoid loss operates solely on image-text pairs and does not require a global view of the pairwise similarities for normalization.
• The sigmoid loss simultaneously allows further scaling up the batch size, while also performing better at smaller batch sizes.
• Combined with Locked-image Tuning, with only four TPUv4 chips, we train a SigLiT model that achieves 84.5% ImageNet zero-shot accuracy in two days.