No Request Left Behind: Tackling Heterogeneity in Long-Context LLM Inference with Medha

Operating System Concepts (9th ed.)

Abraham Silberschatz, Peter B. Galvin, and Greg Gagne. 2012 · 2012

Earlier work this paper cites.

Attention is All you Need. In Advances in Neural Information Processing Systems , I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.), Vol. 30. Curran Associates, Inc

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017 · 2017

Earlier work this paper cites.

Gpipe: Efficient training of giant neural networks using pipeline parallelism

Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al · 2019

Earlier work this paper cites.

Megatron-LM: Training multi-billion parameter language models using gpu model parallelism

Original

Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019 · 2019

Earlier work this paper cites.

Sequence parallelism: Long sequence training from system perspective

Original

Shenggui Li, Fuzhao Xue, Chaitanya Baranwal, Yongbin Li, and Yang You. 2021a · 2021

Earlier work this paper cites.

TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Models

Original

Zhuohan Li, Siyuan Zhuang, Shiyuan Guo, Danyang Zhuo, Hao Zhang, Dawn Song, and Ion Stoica. 2021b · 2021

Earlier work this paper cites.

Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedings of the international conference for high performance computing, networking, storage and analysis . 1–15

Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, et al · 2021

Earlier work this paper cites.

Orca: A Distributed Serving System for Transformer-Based Generative Models. In OSDI

Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022 · 2022

Earlier work this paper cites.

SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills

Original

Amey Agrawal, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, and Ramachandran Ramjee. 2023 · 2023

Earlier work this paper cites.

L-eval: Instituting standardized evaluation for long context language models

Original

Chenxin An, Shansan Gong, Ming Zhong, Xingjian Zhao, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. 2023 · 2023

Earlier work this paper cites.

Operating Systems: Three Easy Pieces

Remzi H. Arpaci-Dusseau and Andrea C. Arpaci-Dusseau. 2023 · 2023

Earlier work this paper cites.

Striped attention: Faster ring attention for causal transformers

Original

William Brandon, Aniruddha Nrusimha, Kevin Qian, Zachary Ankner, Tian Jin, Zhiye Song, and Jonathan Ragan-Kelley. 2023 · 2023

Earlier work this paper cites.

PaLM: Scaling Language Modeling with Pathways

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al · 2023

Earlier work this paper cites.

FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning

Original

Tri Dao. 2023 · 2023

Earlier work this paper cites.

Efficient Memory Management for Large Language Model Serving with PagedAttention. In SOSP

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023 · 2023

Cited alongside, same era.

Ring Attention with Blockwise Transformers for Near-Infinite Context

Original

Hao Liu, Matei Zaharia, and Pieter Abbeel. 2023 · 2023

Cited alongside, same era.

POLCA: Power Oversubscription in LLM Cloud Providers

Original

Pratyush Patel, Esha Choukse, Chaojie Zhang, Íñigo Goiri, Brijesh Warrier, Nithish Mahalingam, and Ricardo Bianchini. 2023 · 2023

Cited alongside, same era.

Fairness in Serving Large Language Models

Original

Ying Sheng, Shiyi Cao, Dacheng Li, Banghua Zhu, Zhuohan Li, Danyang Zhuo, Joseph E Gonzalez, and Ion Stoica. 2023 · 2023

Cited alongside, same era.

OpenChat: Advancing Open-source Language Models with Mixed-Quality Data

Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction. In The 5th International Workshop on Cloud Intelligence / AIOps at ASPLOS 2024

Haoran Qiu, Weichao Mao, Archit Patke, Shengkun Cui, Saurabh Jha, Chen Wang, Hubertus Franke, Zbigniew T. Kalbarczyk, Tamer Başar, and Ravishankar K. Iyer. 2024a · 2024

Closest in time.

Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

Original

M Reid, N Savinov, D Teplyashin, Lepikhin Dmitry, T Lillicrap, JB Alayrac, R Soricut, A Lazaridou, O Firat, et al · 2024

Closest in time.

Roformer: Enhanced transformer with rotary position embedding

Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024 · 2024

Closest in time.

Llumnix: Dynamic Scheduling for Large Language Model Serving. In OSDI

Biao Sun, Ziming Huang, Hanyu Zhao, Wencong Xiao, Xinyi Zhang, Yong Li, and Wei Lin. 2024 · 2024

Closest in time.

LoongServe: Efficiently Serving Long-context Large Language Models with Elastic Sequence Parallelism. In SOSP