Fetching the paper…

Reading the bibliography…

2024

WDMoE: Wireless Distributed Mixture of Experts for Large Language Models

Xue, Nan, Sun, Yaping, Chen, Zhiyong et al.

Original
PDF
Post on X

Understand

Large Language Models (LLMs) have achieved significant success in various natural language processing tasks, but the role of wireless networks in supporting LLMs has not been thoroughly explored.

In this paper, we propose a wireless distributed Mixture of Experts (WDMoE) architecture to enable collaborative deployment of LLMs across edge servers at the base station (BS) and mobile devices in wireless networks.
Specifically, we decompose the MoE layer in LLMs by placing the gating network and the preceding neural network layer at BS, while distributing the expert networks among the devices.
This deployment leverages the parallel inference capabilities of expert networks on mobile devices, effectively utilizing the limited computing and caching resources of these devices.

WDMoE: Wireless Distributed Mixture of Experts for Large Language Models · Around

Built on

T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal et al. , “Language Models are Few-Shot Learners,” in Advances in Neural Information Processing Systems , vol. 33, 2020, pp. 1877–1901
1901
Earlier work this paper cites.
R. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton, “Adaptive Mixtures of Local Experts,” Neural Computation , vol. 3, no. 1, pp. 79–87, 1991
1991
Earlier work this paper cites.
S. Dempe, Foundations of bilevel programming . Springer Science & Business Media, 2002
2002
Earlier work this paper cites.
S. Boyd and L. Vandenberghe, Convex Optimization . Cambridge university press, 2004
2004
Earlier work this paper cites.
H. Liu, Z. Chen, and L. Qian, “The three primary colors of mobile systems,” IEEE Communications Magazine , vol. 54, no. 9, pp. 15–21, 2016
2016
Earlier work this paper cites.
N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean, “Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer,” in International Conference on Learning Representations , 2017
2017
Earlier work this paper cites.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is All you Need,” in Advances in Neural Information Processing Systems , vol. 30, 2017
2017
Earlier work this paper cites.
P. Molchanov, S. Tyree, T. Karras, T. Aila, and J. Kautz, “Pruning Convolutional Neural Networks for Resource Efficient Inference,” in International Conference on Learning Representations , 2017
2017
Earlier work this paper cites.
A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever, “Improving Language Understanding by Generative Pre-Training,” 2018
2018
Earlier work this paper cites.
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” arXiv preprint arXiv:1810.04805 , 2018
Original
2018
Earlier work this paper cites.
P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord, “Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge,” arXiv preprint arXiv:1803.05457 , 2018
Original
2018
Earlier work this paper cites.
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Language Models Are Unsupervised Multitask Learners,” OpenAI blog , vol. 1, no. 8, p. 9, 2019
2019
Earlier work this paper cites.
“IEEE Standard for Floating-Point Arithmetic,” IEEE Std 754-2019 (Revision of IEEE 754-2008) , pp. 1–84, 2019
2019
Earlier work this paper cites.
C. Clark, K. Lee, M.-W. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova, “BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions,” arXiv preprint arXiv:1905.10044 , 2019
Original
2019
Earlier work this paper cites.
J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, “Scaling Laws for Neural Language Models,” arXiv preprint arXiv:2001.08361 , 2020
Original
2020
Earlier work this paper cites.
P. Virtanen, R. Gommers, T. E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau et al. , “SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python,” Nature Methods , vol. 17, pp. 261–272, 2020
2020
Earlier work this paper cites.
Y. Bisk, R. Zellers, R. L. Bras, J. Gao, and Y. Choi, “PIQA: Reasoning about Physical Commonsense in Natural Language,” in Proceedings of the AAAI conference on artificial intelligence , vol. 34, no. 05, 2020, pp. 7432–7439
2020
Earlier work this paper cites.

Similar

D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen, “GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding,” in International Conference on Learning Representations , 2021
2021
Cited alongside, same era.
D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, “Measuring Massive Multitask Language Understanding,” in International Conference on Learning Representations , 2021
2021
Cited alongside, same era.
M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan et al. , “Evaluating Large Language Models Trained on Code,” arXiv preprint arXiv:2107.03374 , 2021
Original
2021
Cited alongside, same era.
K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman, “Training Verifiers to Solve Math Word Problems,” arXiv preprint arXiv:2110.14168 , 2021
Original
2021
Cited alongside, same era.

Then

H. Zhou, C. Hu, Y. Yuan, Y. Cui, Y. Jin, C. Chen, H. Wu, D. Yuan, L. Jiang, D. Wu, X. Liu, C. Zhang, X. Wang, and J. Liu, “Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunities,” arXiv preprint arXiv:2405.10825 , 2024
Original
2024
Closest in time.
N. Sehad, L. Bariah, W. Hamidouche, H. Hellaoui, R. Jäntti, and M. Debbah, “Generative AI for Immersive Communication: The Next Frontier in Internet-of-Senses Through 6G,” arXiv preprint arXiv:2404.01713 , 2024
Original
2024
Closest in time.
J. Xu, Z. Li, W. Chen, Q. Wang, X. Gao, Q. Cai, and Z. Ling, “On-Device Language Models: A Comprehensive Review,” arXiv preprint arXiv:2409.00088 , 2024
Original
2024
Closest in time.
J. Lin, J. Tang, H. Tang, S. Yang, W.-M. Chen, W.-C. Wang, G. Xiao, X. Dang, C. Gan, and S. Han, “AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration,” Proceedings of Machine Learning and Systems , vol. 6, pp. 87–100, 2024
2024

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, and C. Sutton, “Program Synthesis with Large Language Models,” arXiv preprint arXiv:2108.07732 , 2021

2021

Cited alongside, same era.

OpenAI. (2022) Introducing ChatGPT. [Online]. Available: https://www.openai.com/blog/chatgpt/

2022

Cited alongside, same era.

J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, E. H. Chi, T. Hashimoto, O. Vinyals, P. Liang, J. Dean, and W. Fedus, “Emergent Abilities of Large Language Models,” Transactions on Machine Learning Research , 2022

2022

Cited alongside, same era.

J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford et al. , “Training Compute-Optimal Large Language Models,” in Proceedings of the 36th International Conference on Neural Information Processing Systems , 2022, pp. 30 016–30 030

2022

Cited alongside, same era.

S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg, H. Nori, H. Palangi, M. T. Ribeiro, and Y. Zhang, “Sparks of Artificial General Intelligence: Early experiments with GPT-4,” arXiv preprint arXiv:2303.12712 , 2023

2023

Cited alongside, same era.

X. Ma, G. Fang, and X. Wang, “LLM-Pruner: On the Structural Pruning of Large Language Models,” Advances in neural information processing systems , vol. 36, pp. 21 702–21 720, 2023

2023

Cited alongside, same era.

R. Pope, S. Douglas, A. Chowdhery, J. Devlin, J. Bradbury, J. Heek, K. Xiao, S. Agrawal, and J. Dean, “Efficiently Scaling Transformer Inference,” Proceedings of Machine Learning and Systems , vol. 5, pp. 606–624, 2023

2023

Cited alongside, same era.

T. Chen, X. Chen, X. Du, A. Rashwan, F. Yang, H. Chen, Z. Wang, and Y. Li, “AdaMV-MoE: Adaptive Multi-Task Vision Mixture-of-Experts,” in 2023 IEEE/CVF International Conference on Computer Vision (ICCV) , 2023, pp. 17 300–17 311

2023

Cited alongside, same era.

Closest in time.

M. Abdin, J. Aneja, H. Awadalla, A. Awadallah, A. A. Awan, N. Bach et al. , “Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone,” arXiv preprint arXiv:2404.14219 , 2024

2024

Closest in time.

T. Gunter, Z. Wang, C. Wang, R. Pang, A. Narayanan, A. Zhang et al. , “Apple Intelligence Foundation Language Models,” arXiv preprint arXiv:2407.21075 , 2024

2024

Closest in time.

OpenAI. (2024) OpenAI and Apple Announce Partnership to Integrate ChatGPT into Apple Experiences. [Online]. Available: https://openai.com/index/openai-and-apple-announce-partnership/

2024

Closest in time.

Y. Tian, Z. Zhang, Y. Yang, Z. Chen, Z. Yang, R. Jin, T. Q. S. Quek, and K.-K. Wong, “An edge-cloud collaboration framework for generative ai service provision with synergetic big cloud model and small edge models,” IEEE Network , vol. 38, no. 5, pp. 37–46, 2024

2024

Closest in time.

H. Wang, H. Li, M. Sheng, and J. Li, “Collaborative fine-tuning of mobile aigc models with wireless channel conditions,” IEEE Wireless Communications , vol. 31, no. 4, pp. 32–38, 2024

2024

Closest in time.

A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman et al. , “The Llama 3 Herd of Models,” arXiv preprint arXiv:2407.21783 , 2024

2024

Closest in time.

A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford et al. , “Mixtral of experts,” arXiv preprint arXiv:2401.04088 , 2024

2024

Closest in time.

Q. Huang, Z. An, N. Zhuang, M. Tao, C. Zhang, Y. Jin, K. Xu, L. Chen, S. Huang, and Y. Feng, “Harder Tasks Need More Experts: Dynamic Routing in MoE Models,” arXiv preprint arXiv:2403.07652 , 2024

2024

Closest in time.

W. Yin, M. Xu, Y. Li, and X. Liu, “LLM as a System Service on Mobile Devices,” arXiv preprint arXiv:2403.11805 , 2024

2024

Closest in time.

Z. Yu, Z. Wang, Y. Li, H. You, R. Gao, X. Zhou, S. R. Bommu, Y. K. Zhao, and Y. C. Lin, “EDGE-LLM: Enabling Efficient Large Language Model Adaptation on Edge Devices via Layerwise Unified Compression and Adaptive Layer Tuning and Voting,” arXiv preprint arXiv:2406.15758 , 2024

2024

Closest in time.

Z. Liu, C. Zhao, F. Iandola, C. Lai, Y. Tian, I. Fedorov, Y. Xiong, E. Chang, Y. Shi, R. Krishnamoorthi, L. Lai, and V. Chandra, “MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases,” arXiv preprint arXiv:2402.14905 , 2024

2024

Closest in time.

D. Peng, Z. Fu, and J. Wang, “PocketLLM: Enabling On-Device Fine-Tuning for Personalized LLMs,” arXiv preprint arXiv:2407.01031 , 2024

2024

Closest in time.