Scaling Laws for Robust Comparison of Open Foundation Language-Vision Models and Datasets

Understand

In studies of transferable learning, scaling laws are obtained for various important foundation models to predict their properties and performance at larger scales.

• We show here how scaling law derivation can also be used for model and dataset comparison, allowing to decide which procedure is to be preferred for pre-training.
• For the first time, full scaling laws based on dense measurements across a wide span of model and samples seen scales are derived for two important language-vision learning procedures, CLIP and MaMMUT, that use either contrastive only or contrastive and captioning text generative loss.
• Ensuring sufficient prediction accuracy for held out points, we use derived scaling laws to compare both models, obtaining evidence for MaMMUT's stronger improvement with scale and better sample efficiency than standard CLIP.