Sub-MoE: Efficient Mixture-of-Expert LLMs Compression via Subspace Expert Merging

Understand

Mixture of Experts (MoE) LLMs face significant obstacles due to their massive parameter scale, which imposes memory, storage, and deployment challenges.

• Although recent expert merging methods promise greater efficiency by consolidating multiple experts, they are fundamentally hindered by parameter conflicts arising from expert specialization.
• In this paper, we present Sub-MoE, a novel MoE compression framework via Subspace Expert Merging.
• Our key insight is to perform joint Singular Value Decomposition (SVD) on concatenated expert weights, reducing conflicting parameters by extracting shared $U$-matrices while enabling effective merging of the expert-specific $V$ components.