Three Towers: Flexible Contrastive Learning with Pretrained Image Models

Understand

We introduce Three Towers (3T), a flexible method to improve the contrastive learning of vision-language models by incorporating pretrained image classifiers.

• While contrastive models are usually trained from scratch, LiT (Zhai et al., 2022) has recently shown performance gains from using pretrained classifier embeddings.
• However, LiT directly replaces the image tower with the frozen embeddings, excluding any potential benefits from training the image tower contrastively.
• With 3T, we propose a more flexible strategy that allows the image tower to benefit from both pretrained embeddings and contrastive training.