Fetching the paper…
Reading the bibliography…
High-demand LLM inference services (e.g., ChatGPT and BARD) support a wide range of requests from short chat conversations to long document reading.
On packet switches with infinite storage
J. Nagle · 1987
Earlier work this paper cites.
Analysis and simulation of a fair queueing algorithm
Alan Demers, Srinivasan Keshav, and Scott Shenker · 1989
Earlier work this paper cites.
Analysis and simulation of a fair queueing algorithm
Alan J. Demers, Srinivasan Keshav, and Scott Shenker · 1989
Earlier work this paper cites.
Stochastic fairness queueing
P.E. McKenney · 1990
Earlier work this paper cites.
A generalized processor sharing approach to flow control in integrated services networks: the single-node case
A.K. Parekh and R.G. Gallager · 1993
Earlier work this paper cites.
A self-clocked fair queueing scheme for broadband applications
S. Jamaloddin Golestani · 1994
Earlier work this paper cites.
Start-time fair queueing: A scheduling algorithm for integrated services packet switching networks
Pawan Goyal, Harrick M. Vin, and Haichen Cheng · 1996
Earlier work this paper cites.
Efficient fair queueing using deficit round-robin
M. Shreedhar and George Varghese · 1996
Earlier work this paper cites.
A proportional share resource allocation algorithm for real-time, time-shared systems
Ion Stoica, Hussein M. Abdel-Wahab, Kevin Jeffay, Sanjoy K. Baruah, Johannes Gehrke, and C. Greg Plaxton · 1996
Earlier work this paper cites.
Hierarchical packet fair queueing algorithms
Jon CR Bennett and Hui Zhang · 1997
Earlier work this paper cites.
Start-time fair queueing: a scheduling algorithm for integrated services packet switching networks
P. Goyal, H.M. Vin, and Haichen Cheng · 1997
Earlier work this paper cites.
Fairness in parallel job scheduling
Uwe Schwiegelshohn and Ramin Yahyapour · 2000
Earlier work this paper cites.
Linux block io—present and future
Jens Axboe · 2004
Earlier work this paper cites.
Interposed proportional sharing for a storage service utility
Wei Jin, Jeffrey S. Chase, and Jasleen Kaur · 2004
Earlier work this paper cites.
Group ratio round-robin: O(1) proportional share scheduling for uniprocessor and multiprocessor systems
Bogdan Caprita, Wong Chun Chan, Jason Nieh, Clifford Stein, and Haoqiang Zheng · 2005
Earlier work this paper cites.
Fair queuing memory systems
Kyle J. Nesbit, Nidhi Aggarwal, James Laudon, and James E. Smith · 2006
Earlier work this paper cites.
A unified framework for max-min and min-max fairness with applications
Bozidar Radunovic and Jean-Yves Le Boudec · 2007
Earlier work this paper cites.
Quincy: Fair scheduling for distributed computing clusters
Michael Isard, Vijayan Prabhakaran, Jon Currey, Udi Wieder, Kunal Talwar, and Andrew Goldberg · 2009
Earlier work this paper cites.
Delay scheduling: a simple technique for achieving locality and fairness in cluster scheduling
Matei Zaharia, Dhruba Borthakur, Joydeep Sen Sarma, Khaled Elmeleegy, Scott Shenker, and Ion Stoica · 2010
Cited alongside, same era.
Dominant resource fairness: fair allocation of multiple resource types
Ali Ghodsi, Matei Zaharia, Benjamin Hindman, Andy Konwinski, Scott Shenker, and Ion Stoica · 2011
Cited alongside, same era.
An introduction to computer networks
Peter L Dorlan · 2016
Cited alongside, same era.
Altruistic scheduling in Multi-Resource clusters
Robert Grandl, Mosharaf Chowdhury, Aditya Akella, and Ganesh Ananthanarayanan · 2016
Cited alongside, same era.
Blockwise parallel decoding for deep autoregressive models
Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit · 2018
Cited alongside, same era.
{ \{ Multi-Queue } \} fair queuing
Mohammad Hedayati, Kai Shen, Michael L Scott, and Mike Marty · 2019
Text generation inference
Hugging Face · 2023
Closest in time.
Text-generation-inference(tgi)
HuggingFace · 2023
Closest in time.
Efficient memory management for large language model serving with pagedattention
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica · 2023
Closest in time.
{ \{ AlpaServe } \} : Statistical multiplexing with model parallelism for deep learning serving
Zhuohan Li, Lianmin Zheng, Yinmin Zhong, Vincent Liu, Ying Sheng, Xin Jin, Yanping Huang, Zhifeng Chen, Hao Zhang, Joseph E Gonzalez, et al · 2023
Closest in time.
Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Rae Ying Yee Wong, Zhuoming Chen, Daiyaan Arfeen, Reyna Abhyankar, and Zhihao Jia · 2023
Closest in time.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Cited alongside, same era.
Balancing efficiency and fairness in heterogeneous gpu clusters for deep learning
Shubham Chaudhary, Ramachandran Ramjee, Muthian Sivathanu, Nipun Kwatra, and Srinidhi Viswanatha · 2020
Cited alongside, same era.
Themis: Fair and efficient gpu cluster scheduling
Kshiteej Mahajan, Arjun Balasubramanian, Arjun Singhvi, Shivaram Venkataraman, Aditya Akella, Amar Phanishayee, and Shuchi Chawla · 2020
Cited alongside, same era.
{ \{ Heterogeneity-Aware } \} cluster scheduling policies for deep learning workloads
Deepak Narayanan, Keshav Santhanam, Fiodar Kazhamiaka, Amar Phanishayee, and Matei Zaharia · 2020
Cited alongside, same era.
Data networks
Dimitri Bertsekas and Robert Gallager · 2021
Cited alongside, same era.
Turbotransformers: an efficient gpu serving system for transformer models
Jiarui Fang, Yang Yu, Chengduo Zhao, and Jie Zhou · 2021
Cited alongside, same era.
Lightseq: A high performance inference library for transformers
Xiaohui Wang, Ying Xiong, Yang Wei, Mingxuan Wang, and Lei Li · 2021
Cited alongside, same era.
Lightllm: Python-based llm inference and serving framework
ModelTC · 2023
Closest in time.
Cheaply estimating inference efficiency metrics for autoregressive transformer models
Deepak Narayanan, Keshav Santhanam, Peter Henderson, Rishi Bommasani, Tony Lee, and Percy Liang · 2023
Closest in time.
Openai api reference
OpenAI · 2023
Closest in time.
Gpt-4 turbo, 2023
OpenAI · 2023
Closest in time.
Rate limit
OpenAI · 2023
Closest in time.
Efficiently scaling transformer inference
Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean · 2023
Closest in time.
S-lora: Serving thousands of concurrent lora adapters
Ying Sheng, Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang, Christopher Chou, Banghua Zhu, Lianmin Zheng, Kurt Keutzer, Joseph E. Gonzalez, and Ion Stoica · 2023
Closest in time.
Flexgen: high-throughput generative inference of large language models with a single gpu
Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang · 2023
Closest in time.
Fast distributed inference serving for large language models
Bingyang Wu, Yinmin Zhong, Zili Zhang, Gang Huang, Xuanzhe Liu, and Xin Jin · 2023
Closest in time.
Lmsys-chat-1m: A large-scale real-world llm conversation dataset
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric Xing, et al · 2023
Closest in time.
Judging llm-as-a-judge with mt-bench and chatbot arena
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al · 2023
Closest in time.
Efficiently programming large language models using sglang
Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Jeff Huang, Chuyue Sun, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al · 2023
Closest in time.