Performance Characterization of Expert Router for Scalable LLM Inference

C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena et al. , “Exploring the limits of transfer learning with a unified text-to-text transformer.” [Online]. Available: http://arxiv.org/abs/1910.10683

Original

1910

Earlier work this paper cites.

V. J. Reddi, C. Cheng, D. Kanter, P. Mattson, G. Schmuelling, C.-J. Wu et al. , “MLPerf inference benchmark.” [Online]. Available: http://arxiv.org/abs/1911.02549

Original

1911

Earlier work this paper cites.

W. Reese, “Nginx: the high-performance web server and reverse proxy,” Linux J. , vol. 2008, no. 173, sep 2008

2008

Earlier work this paper cites.

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel et al. , “Scikit-learn: Machine learning in Python,” vol. 12, pp. 2825–2830, 2011

2011

Earlier work this paper cites.

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez et al. , “Attention is all you need,” in Advances in Neural Information Processing Systems , I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds., vol. 30. Curran Associates, Inc., 2017

2017

Earlier work this paper cites.

D. Buchaca, J. L. Berral, C. Wang, and A. Youssef, “Proactive container auto-scaling for cloud native machine learning services,” in 2020 IEEE 13th International Conference on Cloud Computing (CLOUD) , 2020, pp. 475–479

2020

Earlier work this paper cites.

R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx et al. , “On the opportunities and risks of foundation models,” 2022

2022

Earlier work this paper cites.

P. Liang, R. Bommasani, T. Lee, D. Tsipras, D. Soylu, M. Yasunaga et al. , “Holistic evaluation of language models,” 2022

2022

Earlier work this paper cites.

G.-I. Yu, J. S. Jeong, G.-W. Kim, S. Kim, and B.-G. Chun, “Orca: A distributed serving system for Transformer-Based generative models,” in 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22) . Carlsbad, CA: USENIX Association, Jul. 2022, pp. 521–538. [Online]. Available: https://www.usenix.org/conference/osdi22/presentation/yu

2022

Cited alongside, same era.

M. R. A. H. Rony, C. Suess, S. R. Bhat, V. Sudhi, J. Schneider, M. Vogel et al. , “Carexpert: Leveraging large language models for in-car conversational question answering,” 2023

2023

Cited alongside, same era.

N. Gruver, A. Sriram, A. Madotto, A. Wilson, C. L. Zitnick, and Z. Ulissi, “Fine-tuned language models generate stable inorganic materials as text,” in AI for Accelerated Materials Design - NeurIPS 2023 Workshop , 2023. [Online]. Available: https://openreview.net/forum?id=0r5DE2ZSwJ

2023

Cited alongside, same era.

Z. Zheng, X. Ren, F. Xue, Y. Luo, X. Jiang, and Y. You, “Response length perception and sequence scheduling: An llm-empowered llm inference pipeline,” in Advances in Neural Information Processing Systems , vol. 36. Curran Associates, Inc., 2023, pp. 65 517–65 530. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2023/file/ce7ff3405c782f761fac7f849b41ae9a-Paper-Conference.pdf

2023

Later among the works it cites.

OpenAI, J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya et al. , “Gpt-4 technical report,” 2024

2024

Closest in time.

Anthropic, “The claude 3 model family: Opus, sonnet, haiku,” https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf , 2024, accessed: 2024-03-31

2024

Closest in time.

A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman et al. , “The llama 3 herd of models,” 2024. [Online]. Available: https://arxiv.org/abs/2407.21783

Original

2024

Closest in time.