Etalon: Holistic Performance Evaluation Framework for LLM Inference Systems

Clipper: A { \{ Low-Latency } \} online prediction serving system

Daniel Crankshaw, Xin Wang, Guilio Zhou, Michael J Franklin, Joseph E Gonzalez, and Ion Stoica · 2017

Earlier work this paper cites.

Tensorflow-serving: Flexible, high-performance ml serving, 2017

Christopher Olston, Noah Fiedel, Kiril Gorovoy, Jeremiah Harmsen, Li Lao, Fangwei Li, Vinu Rajashekhar, Sukriti Ramesh, and Jordan Soyke · 2017

Earlier work this paper cites.

A discourse-aware attention model for abstractive summarization of long documents

Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian · 2018

Earlier work this paper cites.

Low latency rnn inference with cellular batching

Pin Gao, Lingfan Yu, Yongwei Wu, and Jinyang Li · 2018

Earlier work this paper cites.

Serving { \{ DNNs } \} like clockwork: Performance predictability from the bottom up

Arpan Gujarati, Reza Karimi, Safya Alzayat, Wei Hao, Antoine Kaufmann, Ymir Vigfusson, and Jonathan Mace · 2020

Earlier work this paper cites.

Soft Real-Time Scheduling

Jeremy P. Erickson and James H. Anderson · 2022

Earlier work this paper cites.

Orca: A distributed serving system for Transformer-Based generative models

Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun · 2022

Cited alongside, same era.

Efficient memory management for large language model serving with pagedattention

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica · 2023

Cited alongside, same era.

Fast inference from transformers via speculative decoding

Yaniv Leviathan, Matan Kalman, and Yossi Matias · 2023

Cited alongside, same era.

Splitwise: Efficient generative llm inference using phase splitting, 2023

Pratyush Patel, Esha Choukse, Chaojie Zhang, Íñigo Goiri, Aashaka Shah, Saeed Maleki, and Ricardo Bianchini · 2023

Cited alongside, same era.

Lmsys-chat-1m: A large-scale real-world llm conversation dataset, 2023

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric. P Xing, Joseph E. Gonzalez, Ion Stoica, and Hao Zhang · 2023

Cited alongside, same era.

Taming throughput-latency tradeoff in llm inference with sarathi-serve

Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S Gulavani, Alexey Tumanov, and Ramachandran Ramjee · 2024

Closest in time.

Taming throughput-latency tradeoff in llm inference with sarathi-serve

Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S Gulavani, Alexey Tumanov, and Ramachandran Ramjee · 2024

Closest in time.

Inference without interference: Disaggregate llm inference for mixed downstream workloads

Original

Cunchen Hu, Heyang Huang, Liangliang Xu, Xusheng Chen, Jiang Xu, Shuang Chen, Hao Feng, Chenxi Wang, Sa Wang, Yungang Bao, et al · 2024

Closest in time.

Mixtral of experts, 2024

Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed · 2024

Closest in time.