Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts

Neural machine translation of rare words with subword units

Original

Rico Sennrich, Barry Haddow, and Alexandra Birch · 2015

Earlier work this paper cites.

Sgdr: Stochastic gradient descent with warm restarts

Ilya Loshchilov and Frank Hutter · 2016

Earlier work this paper cites.

Outrageously large neural networks: The sparsely-gated mixture-of-experts layer

Original

Noam M. Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E. Hinton, and Jeff Dean · 2017

Earlier work this paper cites.

Attention is all you need

Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin · 2017

Earlier work this paper cites.

Gshard: Scaling giant models with conditional computation and automatic sharding

Original

Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam M. Shazeer, and Z. Chen · 2020

Cited alongside, same era.

Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity

William Fedus, Barret Zoph, and Noam M. Shazeer · 2021

Cited alongside, same era.

Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models

Original

Damai Dai, Chengqi Deng, Chenggang Zhao, Runxin Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Yu Wu, Zhenda Xie, Y. K. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang · 2024

Cited alongside, same era.

Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence

Original

DeepSeek-AI, Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y. Wu, Yukun Li, Huazuo Gao, Shirong Ma, Wangding Zeng, Xiao Bi, Zihui Gu, Hanwei Xu, Damai Dai, Kai Dong, Liyue Zhang, Yishi Piao, Zhibin Gou, Zhenda Xie, Zhewen Hao, Bing-Li Wang, Jun-Mei Song, Deli Chen, Xin Xie, Kang Guan, Yu mei You, Aixin Liu, Qiushi Du, Wenjun Gao, Xuan Lu, Qinyu Chen, Yaohui Wang, Chengqi Deng, Jiashi Li, Chenggang Zhao, Chong Ruan, Fuli Luo, and Wenfeng Liang · 2024

Closest in time.

Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model

Original

Zhihong Shao, Damai Dai, Daya Guo, Bo Liu, and Zihan Wang · 2024

Closest in time.

Mixture-of-experts with expert choice routing

Original

Yan-Quan Zhou, Tao Lei, Han-Chu Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M. Dai, Zhifeng Chen, Quoc V. Le, and James Laudon · 2024

Closest in time.