Statistical Perspective of Top-K Sparse Softmax Gating Mixture of Experts

Understand

Top-K sparse softmax gating mixture of experts has been widely used for scaling up massive deep-learning architectures without increasing the computational cost.

• Despite its popularity in real-world applications, the theoretical understanding of that gating function has remained an open problem.
• The main challenge comes from the structure of the top-K sparse softmax gating function, which partitions the input space into multiple regions with distinct behaviors.
• By focusing on a Gaussian mixture of experts, we establish theoretical results on the effects of the top-K sparse softmax gating function on both density and parameter estimations.