OptMerge: Unifying Multimodal LLM Capabilities and Modalities via Model Merging

Understand

Foundation models update slowly due to resource-intensive training, whereas domain-specific models evolve rapidly between releases.

• Model merging seeks to combine multiple expert models into a single, more capable model, reducing storage and serving costs while supporting decentralized development.
• Despite its potential, previous studies have primarily focused on merging visual classification models or Large Language Models (LLMs) for code and math tasks.
• Recently, Multimodal LLMs (MLLMs) that extend LLMs through large-scale multimodal training have gained traction.