Expert Merging in Sparse Mixture of Experts with Nash Bargaining

Understand

Existing expert merging strategies for Sparse Mixture of Experts (SMoE) typically rely on input-dependent or input-independent averaging of expert parameters, but often lack a principled weighting mechanism.

• In this work, we reinterpret expert merging through the lens of game theory, revealing cooperative and competitive dynamics among experts.
• Based on this perspective, we introduce Nash Merging of Experts (NAMEx), a novel framework that incorporates Nash Bargaining into the merging process, enabling more balanced and efficient collaboration among experts.
• Additionally, we incorporate complex momentum into NAMEx to accelerate expert propagation with theoretical guarantees for convergence.