MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training

Understand

In this work, we discuss building performant Multimodal Large Language Models (MLLMs).

• In particular, we study the importance of various architecture components and data choices.
• Through careful and comprehensive ablations of the image encoder, the vision language connector, and various pre-training data choices, we identified several crucial design lessons.
• For example, we demonstrate that for large-scale multimodal pre-training using a careful mix of image-caption, interleaved image-text, and text-only data is crucial for achieving state-of-the-art (SOTA) few-shot results across multiple benchmarks, compared to other published pre-training results.