Scaling Pre-training to One Hundred Billion Data for Vision Language Models

Understand

We provide an empirical investigation of the potential of pre-training vision-language models on an unprecedented scale: 100 billion examples.

• We find that model performance tends to saturate at this scale on many common Western-centric classification and retrieval benchmarks, such as COCO Captions.
• Nevertheless, tasks of cultural diversity achieve more substantial gains from the 100-billion scale web data, thanks to its coverage of long-tail concepts.
• Furthermore, we analyze the model's multilinguality and show gains in low-resource languages as well.