Finding Fantastic Experts in MoEs: A Unified Study for Expert Dropping Strategies and Observations

Glam: Efficient scaling of language models with mixture-of-experts

Du, N., Huang, Y., Dai, A. M., Tong, S., Lepikhin, D., Xu, Y., Krikun, M., Zhou, Y., Yu, A. W., Firat, O., et al · 2022

Cited alongside, same era.

Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity

Fedus, W., Zoph, B., and Shazeer, N · 2022

Cited alongside, same era.

Gptq: Accurate post-training quantization for generative pre-trained transformers

Original

Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D · 2022

Cited alongside, same era.

Training your sparse neural network better with any mask

Jaiswal, A. K., Ma, H., Chen, T., Ding, Y., and Wang, Z · 2022

Cited alongside, same era.

Is a modular architecture enough?

Mittal, S., Bengio, Y., and Lajoie, G · 2022

Cited alongside, same era.

Unmasking the lottery ticket hypothesis: What’s encoded in a winning ticket’s mask?, 2022

Original

Paul, M., Chen, F., Larsen, B. W., Frankle, J., Ganguli, S., and Dziugaite, G. K · 2022

Cited alongside, same era.

Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation ai scale, 2022

Original

Rajbhandari, S., Li, C., Yao, Z., Zhang, M., Aminabadi, R. Y., Awan, A. A., Rasley, J., and He, Y · 2022

Cited alongside, same era.

Structural pruning via latency-saliency knapsack, 2022

Original

Shen, M., Yin, H., Molchanov, P., Mao, L., Liu, J., and Alvarez, J. M · 2022

Cited alongside, same era.

Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model, 2024

Original

DeepSeek-AI, Liu, A., Feng, B., Wang, B., Wang, B., Liu, B., Zhao, C., Dengr, C., Ruan, C., Dai, D., Guo, D., Yang, D., Chen, D., Ji, D., Li, E., Lin, F., Luo, F., Hao, G., Chen, G., Li, G., Zhang, H., Xu, H., Yang, H., Zhang, H., Ding, H., Xin, H., Gao, H., Li, H., Qu, H., Cai, J. L., Liang, J., Guo, J., Ni, J., Li, J., Chen, J., Yuan, J., Qiu, J., Song, J., Dong, K., Gao, K., Guan, K., Wang, L., Zhang, L., Xu, L., Xia, L., Zhao, L., Zhang, L., Li, M., Wang, M., Zhang, M., Zhang, M., Tang, M., Li, M., Tian, N., Huang, P., Wang, P., Zhang, P., Zhu, Q., Chen, Q., Du, Q., Chen, R. J., Jin, R. L., Ge, R., Pan, R., Xu, R., Chen, R., Li, S. S., Lu, S., Zhou, S., Chen, S., Wu, S., Ye, S., Ma, S., Wang, S., Zhou, S., Yu, S., Zhou, S., Zheng, S., Wang, T., Pei, T., Yuan, T., Sun, T., Xiao, W. L., Zeng, W., An, W., Liu, W., Liang, W., Gao, W., Zhang, W., Li, X. Q., Jin, X., Wang, X., Bi, X., Liu, X., Wang, X., Shen, X., Chen, X., Chen, X., Nie, X., Sun, X., Wang, X., Liu, X., Xie, X., Yu, X., Song, X., Zhou, X., Yang, X., Lu, X., Su, X., Wu, Y., Li, Y. K., Wei, Y. X., Zhu, Y. X., Xu, Y., Huang, Y., Li, Y., Zhao, Y., Sun, Y., Li, Y., Wang, Y., Zheng, Y., Zhang, Y., Xiong, Y., Zhao, Y., He, Y., Tang, Y., Piao, Y., Dong, Y., Tan, Y., Liu, Y., Wang, Y., Guo, Y., Zhu, Y., Wang, Y., Zou, Y., Zha, Y., Ma, Y., Yan, Y., You, Y., Liu, Y., Ren, Z. Z., Ren, Z., Sha, Z., Fu, Z., Huang, Z., Zhang, Z., Xie, Z., Hao, Z., Shao, Z., Wen, Z., Xu, Z., Zhang, Z., Li, Z., Wang, Z., Gu, Z., Li, Z., and Xie, Z · 2024

Later among the works it cites.

Demystifying the compression of mixture-of-experts through a unified framework

Original

He, S., Dong, D., Ding, L., and Li, A · 2024

Later among the works it cites.

Decoding compressed trust: Scrutinizing the trustworthiness of efficient llms under compression

Original

Hong, J., Duan, J., Zhang, C., Li, Z., Xie, C., Lieberman, K., Diffenderfer, J., Bartoldson, B., Jaiswal, A., Xu, K., et al · 2024

Later among the works it cites.

From galore to welore: How low-rank weights non-uniformly emerge from low-rank gradients

Original

Jaiswal, A., Yin, L., Zhang, Z., Liu, S., Zhao, J., Tian, Y., and Wang, Z · 2024

Later among the works it cites.

Mixtral of experts

Original

Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. d. l., Hanna, E. B., Bressand, F., et al · 2024

Later among the works it cites.

Mlp can be a good transformer learner

Lin, S., Lyu, P., Liu, D., Tang, T., Liang, X., Song, A., and Chang, X · 2024

Later among the works it cites.

Not all experts are equal: Efficient expert pruning and skipping for mixture-of-experts large language models

Original

Lu, X., Liu, Q., Xu, Y., Zhou, A., Huang, S., Zhang, B., Yan, J., and Li, H · 2024

Later among the works it cites.

Seer-moe: Sparse expert efficiency through regularization for mixture-of-experts

Original

Muzio, A., Sun, A., and He, C · 2024

Later among the works it cites.

Revisiting smoe language models by evaluating inefficiencies with task specific expert pruning, 2024

Original

Sarkar, S., Lausen, L., Cevher, V., Zha, S., Brox, T., and Karypis, G · 2024

Later among the works it cites.

Smoothquant: Accurate and efficient post-training quantization for large language models, 2024

Original

Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S · 2024

Later among the works it cites.

Q-galore: Quantized galore with int4 projection and layer-adaptive low-rank gradients

Original

Zhang, Z., Jaiswal, A., Yin, L., Liu, S., Zhao, J., Tian, Y., and Wang, Z · 2024

Later among the works it cites.

Galore: Memory-efficient llm training by gradient low-rank projection

Original

Zhao, J., Zhang, Z., Chen, B., Wang, Z., Anandkumar, A., and Tian, Y · 2024

Later among the works it cites.

Idea prune: An integrated enlarge-and-prune pipeline in generative language model pretraining

Li, Y., Du, X., Jaiswal, A., Lei, T., Zhao, T., Wang, C., and Wang, J · 2025

Closest in time.