Fetching the paper…

Reading the bibliography…

2024

LLM Inference Serving: Survey of Recent Advances and Opportunities

Li, Baolin, Jiang, Yankai, Gadepally, Vijay et al.

Original
PDF
Post on X

Understand

This survey offers a comprehensive overview of recent advancements in Large Language Model (LLM) serving systems, focusing on research since the year 2023.

We specifically examine system-level enhancements that improve performance and efficiency without altering the core LLM decoding mechanisms.
By selecting and reviewing high-quality papers from prestigious ML and system venues, we highlight key innovations and practical considerations for deploying and scaling LLMs in real-world production environments.
This survey serves as a valuable resource for LLM practitioners seeking to stay abreast of the latest developments in this rapidly evolving field.

LLM Inference Serving: Survey of Recent Advances and Opportunities · Around

Built on

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems , vol. 30, 2017
2017
Earlier work this paper cites.
A. Radford, K. Narasimhan, T. Salimans, I. Sutskever et al. , “Improving language understanding by generative pre-training,” 2018
2018
Earlier work this paper cites.
P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel et al. , “Retrieval-augmented generation for knowledge-intensive nlp tasks,” Advances in Neural Information Processing Systems , vol. 33, pp. 9459–9474, 2020
2020
Earlier work this paper cites.
G.-I. Yu, J. S. Jeong, G.-W. Kim, S. Kim, and B.-G. Chun, “Orca: A distributed serving system for { \{ Transformer-Based } \} generative models,” in 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22) , 2022, pp. 521–538
2022
Earlier work this paper cites.
X. Miao, G. Oliaro, Z. Zhang, X. Cheng, H. Jin, T. Chen, and Z. Jia, “Towards efficient generative large language model serving: A survey from algorithms to systems,” arXiv preprint arXiv:2312.15234 , 2023
Original
2023
Earlier work this paper cites.
R. Pope, S. Douglas, A. Chowdhery, J. Devlin, J. Bradbury, J. Heek, K. Xiao, S. Agrawal, and J. Dean, “Efficiently scaling transformer inference,” Proceedings of Machine Learning and Systems , vol. 5, pp. 606–624, 2023
2023
Earlier work this paper cites.
W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with pagedattention,” in Proceedings of the 29th Symposium on Operating Systems Principles , 2023, pp. 611–626
2023
Earlier work this paper cites.
H. Liu, M. Zaharia, and P. Abbeel, “Ring attention with blockwise transformers for near-infinite context,” arXiv preprint arXiv:2310.01889 , 2023
Original
2023
Earlier work this paper cites.
Y. Sheng, L. Zheng, B. Yuan, Z. Li, M. Ryabinin, B. Chen, P. Liang, C. Ré, I. Stoica, and C. Zhang, “Flexgen: High-throughput generative inference of large language models with a single gpu,” in International Conference on Machine Learning . PMLR, 2023, pp. 31 094–31 116
2023
Earlier work this paper cites.
Y. Jin, C.-F. Wu, D. Brooks, and G.-Y. Wei, “S3: Increasing gpu utilization during generative inference for higher throughput,” Advances in Neural Information Processing Systems , vol. 36, pp. 18 015–18 027, 2023
2023
Earlier work this paper cites.
P. Patel, E. Choukse, C. Zhang, Í. Goiri, A. Shah, S. Maleki, and R. Bianchini, “Splitwise: Efficient generative llm inference using phase splitting,” arXiv preprint arXiv:2311.18677 , 2023
Original
2023
Earlier work this paper cites.
J. Li, Y. Jiang, Y. Zhu, C. Wang, and H. Xu, “Accelerating distributed { \{ MoE } \} training and inference with lina,” in 2023 USENIX Annual Technical Conference (USENIX ATC 23) , 2023, pp. 945–959
2023
Earlier work this paper cites.
H. Huang, N. Ardalani, A. Sun, L. Ke, H.-H. S. Lee, A. Sridhar, S. Bhosale, C.-J. Wu, and B. Lee, “Towards moe deployment: Mitigating inefficiencies in mixture-of-expert (moe) inference,” arXiv preprint arXiv:2303.06182 , 2023
Original
2023
Earlier work this paper cites.
Y. Sheng, S. Cao, D. Li, B. Zhu, Z. Li, D. Zhuo, J. E. Gonzalez, and I. Stoica, “Fairness in serving large language models,” arXiv preprint arXiv:2401.00588 , 2023
Original
2023
Earlier work this paper cites.
L. Chen, M. Zaharia, and J. Zou, “Frugalgpt: How to use large language models while reducing cost and improving performance,” arXiv preprint arXiv:2305.05176 , 2023
Original
2023
Earlier work this paper cites.
T. Cai, Y. Li, Z. Geng, H. Peng, J. D. Lee, D. Chen, and T. Dao, “Medusa: Simple llm inference acceleration framework with multiple decoding heads,” arXiv preprint arXiv:2401.10774 , 2024
Original
2024
Earlier work this paper cites.
Y. Fu, P. Bailis, I. Stoica, and H. Zhang, “Break the sequential dependency of llm inference using lookahead decoding,” arXiv preprint arXiv:2402.02057 , 2024
Original
2024
Earlier work this paper cites.
M. Adnan, A. Arunkumar, G. Jain, P. Nair, I. Soloveychik, and P. Kamath, “Keyformer: Kv cache reduction through key tokens selection for efficient generative inference,” Proceedings of Machine Learning and Systems , vol. 6, pp. 114–127, 2024
2024
Earlier work this paper cites.
Z. Yuan, Y. Shang, Y. Zhou, Z. Dong, C. Xue, B. Wu, Z. Li, Q. Gu, Y. J. Lee, Y. Yan et al. , “Llm inference unveiled: Survey and roofline model insights,” arXiv preprint arXiv:2402.16363 , 2024
Original
2024
Earlier work this paper cites.
Z. Zhou, X. Ning, K. Hong, T. Fu, J. Xu, S. Li, Y. Lou, L. Wang, Z. Yuan, X. Li et al. , “A survey on efficient inference for large language models,” arXiv preprint arXiv:2404.14294 , 2024
Original
2024
Earlier work this paper cites.

Similar

R. Prabhu, A. Nayak, J. Mohan, R. Ramjee, and A. Panwar, “vattention: Dynamic memory management for serving llms without pagedattention,” arXiv preprint arXiv:2405.04437 , 2024
Original
2024
Cited alongside, same era.
I. Gim, G. Chen, S.-s. Lee, N. Sarda, A. Khandelwal, and L. Zhong, “Prompt cache: Modular attention reuse for low-latency inference,” Proceedings of Machine Learning and Systems , vol. 6, pp. 325–338, 2024
2024
Cited alongside, same era.
B. Gao, Z. He, P. Sharma, Q. Kang, D. Jevdjic, J. Deng, X. Yang, Z. Yu, and P. Zuo, “Cost-Efficient large language model serving for multi-turn conversations with CachedAttention,” in 2024 USENIX Annual Technical Conference (USENIX ATC 24) . Santa Clara, CA: USENIX Association, Jul. 2024, pp. 111–126. [Online]. Available: https://www.usenix.org/conference/atc24/presentation/gao-bin-cost
2024
Cited alongside, same era.
B. Lin, T. Peng, C. Zhang, M. Sun, L. Li, H. Zhao, W. Xiao, Q. Xu, X. Qiu, S. Li et al. , “Infinite-llm: Efficient llm service for long context with distattention and distributed kvcache,” arXiv preprint arXiv:2401.02669 , 2024

Then

T. Griggs, X. Liu, J. Yu, D. Kim, W.-L. Chiang, A. Cheung, and I. Stoica, “M \ \backslash ’elange: Cost efficient large language model serving by exploiting gpu heterogeneity,” arXiv preprint arXiv:2404.14527 , 2024
Original
2024
Closest in time.
B. Sun, Z. Huang, H. Zhao, W. Xiao, X. Zhang, Y. Li, and W. Lin, “Llumnix: Dynamic scheduling for large language model serving,” arXiv preprint arXiv:2406.03243 , 2024
Original
2024
Closest in time.
P. Patel, E. Choukse, C. Zhang, Í. Goiri, B. Warrier, N. Mahalingam, and R. Bianchini, “Characterizing power management opportunities for llms in the cloud,” in Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3 , 2024, pp. 207–222
2024
Closest in time.
Z. Yang, Y. Yang, C. Zhao, Q. Guo, W. He, and W. Ji, “Perllm: Personalized inference scheduling with edge-cloud collaboration for diverse llm services,” arXiv preprint arXiv:2405.14636 , 2024
Original

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

2024

Cited alongside, same era.

C. Hu, H. Huang, J. Hu, J. Xu, X. Chen, T. Xie, C. Wang, S. Wang, Y. Bao, N. Sun et al. , “Memserve: Context caching for disaggregated llm serving with elastic memory pool,” arXiv preprint arXiv:2406.17565 , 2024

2024

Cited alongside, same era.

W. Lee, J. Lee, J. Seo, and J. Sim, “Infinigen: Efficient generative inference of large language models with dynamic kv cache management,” arXiv preprint arXiv:2406.19707 , 2024

2024

Cited alongside, same era.

B. Wu, S. Liu, Y. Zhong, P. Sun, X. Liu, and X. Jin, “Loongserve: Efficiently serving long-context large language models with elastic sequence parallelism,” arXiv preprint arXiv:2404.09526 , 2024

2024

Cited alongside, same era.

Z. Liu, J. Yuan, H. Jin, S. Zhong, Z. Xu, V. Braverman, B. Chen, and X. Hu, “Kivi: A tuning-free asymmetric 2bit quantization for kv cache,” arXiv preprint arXiv:2402.02750 , 2024

2024

Cited alongside, same era.

H. Kang, Q. Zhang, S. Kundu, G. Jeong, Z. Liu, T. Krishna, and T. Zhao, “Gear: An efficient kv cache compression recipefor near-lossless generative inference of llm,” arXiv preprint arXiv:2403.05527 , 2024

2024

Cited alongside, same era.

A. Liu, J. Liu, Z. Pan, Y. He, G. Haffari, and B. Zhuang, “Minicache: Kv cache compression in depth dimension for large language models,” arXiv preprint arXiv:2405.14366 , 2024

2024

Cited alongside, same era.

Z. Zheng, X. Ren, F. Xue, Y. Luo, X. Jiang, and Y. You, “Response length perception and sequence scheduling: An llm-empowered llm inference pipeline,” Advances in Neural Information Processing Systems , vol. 36, 2024

2024

Cited alongside, same era.

C. Holmes, M. Tanaka, M. Wyatt, A. A. Awan, J. Rasley, S. Rajbhandari, R. Y. Aminabadi, H. Qin, A. Bakhtiari, L. Kurilenko et al. , “Deepspeed-fastgen: High-throughput text generation for llms via mii and deepspeed-inference,” arXiv preprint arXiv:2401.08671 , 2024

2024

Cited alongside, same era.

2024

Closest in time.

X. Miao, G. Oliaro, X. Cheng, M. Wu, C. Unger, and Z. Jia, “Flexllm: A system for co-serving large language model inference and parameter-efficient finetuning,” arXiv preprint arXiv:2402.18789 , 2024

2024

Closest in time.

J. Liu, Z. Wu, J.-W. Chung, F. Lai, M. Lee, and M. Chowdhury, “Andes: Defining and enhancing quality-of-experience in llm-based text streaming services,” arXiv preprint arXiv:2404.16283 , 2024

2024

Closest in time.

Y. Zhu, J.-C. Gu, C. Sikora, H. Ko, Y. Liu, C.-C. Lin, L. Shu, L. Luo, L. Meng, B. Liu et al. , “Accelerating inference of retrieval-augmented generation via sparse context selection,” arXiv preprint arXiv:2405.16178 , 2024

2024

Closest in time.

C. Jin, Z. Zhang, X. Jiang, F. Liu, X. Liu, X. Liu, and X. Jin, “Ragcache: Efficient knowledge caching for retrieval-augmented generation,” arXiv preprint arXiv:2404.12457 , 2024

2024

Closest in time.

J. Yao, H. Li, Y. Liu, S. Ray, Y. Cheng, Q. Zhang, K. Du, S. Lu, and J. Jiang, “Cacheblend: Fast large language model serving with cached knowledge fusion,” arXiv preprint arXiv:2405.16444 , 2024

2024

Closest in time.

J. Yao, Q. Anthony, A. Shafi, H. Subramoni, and D. K. D. Panda, “Exploiting inter-layer expert affinity for accelerating mixture-of-experts model inference,” in 2024 IEEE International Parallel and Distributed Processing Symposium (IPDPS) . IEEE, 2024, pp. 915–925

2024

Closest in time.

Z. Du, S. Li, Y. Wu, X. Jiang, J. Sun, Q. Zheng, Y. Wu, A. Li, H. Li, and Y. Chen, “Sida: Sparsity-inspired data-aware serving for efficient and scalable large mixture-of-experts models,” Proceedings of Machine Learning and Systems , vol. 6, pp. 224–238, 2024

2024

Closest in time.

L. Xue, Y. Fu, Z. Lu, L. Mai, and M. Marina, “Moe-infinity: Activation-aware expert offloading for efficient moe serving,” arXiv preprint arXiv:2401.14361 , 2024

2024

Closest in time.

K. Kamahori, Y. Gu, K. Zhu, and B. Kasikci, “Fiddler: Cpu-gpu orchestration for fast inference of mixture-of-experts models,” arXiv preprint arXiv:2402.07033 , 2024

2024

Closest in time.

B. Li, Y. Jiang, V. Gadepally, and D. Tiwari, “Toward sustainable genai using generation directives for carbon-friendly large language model inference,” arXiv preprint arXiv:2403.12900 , 2024

2024

Closest in time.

K. Hong, G. Dai, J. Xu, Q. Mao, X. Li, J. Liu, Y. Dong, Y. Wang et al. , “Flashdecoding++: Faster large language model inference with asynchronization, flat gemm optimization, and heuristics,” Proceedings of Machine Learning and Systems , vol. 6, pp. 148–161, 2024

2024

Closest in time.

C. Lin, Z. Han, C. Zhang, Y. Yang, F. Yang, C. Chen, and L. Qiu, “Parrot: Efficient serving of llm-based applications with semantic variable,” arXiv preprint arXiv:2405.19888 , 2024

2024

Closest in time.

J. Shah, G. Bikshandi, Y. Zhang, V. Thakkar, P. Ramani, and T. Dao, “Flashattention-3: Fast and accurate attention with asynchrony and low-precision,” arXiv preprint arXiv:2407.08608 , 2024

2024

Closest in time.

X. Miao, G. Oliaro, Z. Zhang, X. Cheng, Z. Wang, Z. Zhang, R. Y. Y. Wong, A. Zhu, L. Yang, X. Shi et al. , “Specinfer: Accelerating large language model serving with tree-based speculative inference and verification,” in Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3 , 2024, pp. 932–949

2024

Closest in time.

I. Ong, A. Almahairi, V. Wu, W.-L. Chiang, T. Wu, J. E. Gonzalez, M. W. Kadous, and I. Stoica, “Routellm: Learning to route llms with preference data,” arXiv preprint arXiv:2406.18665 , 2024

2024

Closest in time.