DynaMo: Runtime Switchable Quantization for MoE with Cross-Dataset Adaptation

Understand

As the Mix-of-Experts (MoE) architecture increases the number of parameters in large models, there is an even greater need for model quantization.

• However, existing quantization methods overlook the expert dynamics of MoE across multiple datasets.
• Moreover, the existing static quantization cannot adapt MoE to various data change scenarios.
• In this paper, we perform a multi-level analysis to reveal MoE dynamics and define the significance of each channel/each expert.