AdapMoE: Adaptive Sensitivity-based Expert Gating and Management for Efficient MoE Inference

Mixture of experts: a literature survey

Saeed Masoudnia and Reza Ebrahimpour. 2014 · 2014

Earlier work this paper cites.

Outrageously large neural networks: The sparsely-gated mixture-of-experts layer

Original

Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017 · 2017

Earlier work this paper cites.

Think you have solved question answering? try arc, the ai2 reasoning challenge

Original

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018 · 2018

Earlier work this paper cites.

Measuring massive multitask language understanding

Original

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020 · 2020

Earlier work this paper cites.

Pyhessian: Neural networks through the lens of the hessian. In 2020 IEEE international conference on big data (Big data) . IEEE, 581–590

Zhewei Yao, Amir Gholami, Kurt Keutzer, and Michael W Mahoney. 2020 · 2020

Earlier work this paper cites.

Zero-infinity: Breaking the gpu memory wall for extreme scale deep learning. In Proceedings of the international conference for high performance computing, networking, storage and analysis . 1–14

Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, and Yuxiong He. 2021 · 2021

Earlier work this paper cites.

{ \{ Zero-offload } \} : Democratizing { \{ billion-scale } \} model training. In 2021 USENIX Annual Technical Conference (USENIX ATC 21) . 551–564

Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, and Yuxiong He. 2021 · 2021

Earlier work this paper cites.

Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale. In SC22: International Conference for High Performance Computing, Networking, Storage and Analysis . IEEE, 1–15

Reza Yazdani Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Minjia Zhang, Jeff Rasley, et al · 2022

Cited alongside, same era.

No language left behind: Scaling human-centered machine translation

Original

Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al · 2022

Cited alongside, same era.

Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity

William Fedus, Barret Zoph, and Noam Shazeer. 2022 · 2022

Cited alongside, same era.

Half-Quadratic Quantization of Large Machine Learning Models

Hicham Badri and Appu Shaji. 2023 · 2023

Cited alongside, same era.

Fast inference of mixture-of-experts language models with offloading

Adaptive gating in mixture-of-experts based language models

Original

Jiamin Li, Qiang Su, Yitao Yang, Yimin Jiang, Cong Wang, and Hong Xu. 2023 · 2023

Later among the works it cites.

Flexgen: High-throughput generative inference of large language models with a single gpu. In International Conference on Machine Learning . PMLR, 31094–31116

Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. 2023 · 2023

Later among the works it cites.

Edgemoe: Fast on-device inference of moe-based large language models

Original

Rongjie Yi, Liwei Guo, Shiyun Wei, Ao Zhou, Shangguang Wang, and Mengwei Xu. 2023 · 2023

Later among the works it cites.

Mixtral of experts

Original

Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al · 2024

Closest in time.