MergePipe: A Budget-Aware Parameter Management System for Scalable LLM Merging

Understand

Large language model (LLM) merging has become a key technique in modern LLM development pipelines, enabling the integration of multiple task- or domain-specific expert models without retraining.

• However, as the number of experts grows, existing merging implementations treat model parameters as unstructured files and execute merges in a stateless, one-shot manner, leading to excessive disk I/O, redundant parameter scans, and poor scalability.
• In this paper, we present \textbf{MergePipe}, a parameter management system for scalable LLM merging.
• MergePipe is the first system that treats LLM merging as a data management and execution problem, and introduces a catalog-driven abstraction over model parameters, merge plans, and execution lineage.