A Practitioner's Guide to Continual Multimodal Pretraining

Understand

Multimodal foundation models serve numerous applications at the intersection of vision and language.

• Still, despite being pretrained on extensive data, they become outdated over time.
• To keep models updated, research into continual pretraining mainly explores scenarios with either (1) infrequent, indiscriminate updates on large-scale new data, or (2) frequent, sample-level updates.
• However, practical model deployment often operates in the gap between these two limit cases, as real-world applications often demand adaptation to specific subdomains, tasks or concepts -- spread over the entire, varying life cycle of a model.