2024

Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving

Qin, Ruoyu, Li, Zheming, He, Weiran et al.

Understand

Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.

It features a KVCache-centric disaggregated architecture that separates the prefill and decoding clusters.
It also leverages the underutilized CPU, DRAM, and SSD resources of the GPU cluster to implement a disaggregated cache of KVCache.
The core of Mooncake is its KVCache-centric scheduler, which balances maximizing overall effective throughput while meeting latency-related Service Level Objectives (SLOs).

Built on

Optimizing gpu energy efficiency with 3d die-stacking graphics memory and reconfigurable memory interface
Jishen Zhao, Guangyu Sun, Gabriel H. Loh, and Yuan Xie · 2013
Earlier work this paper cites.
Neurocube: A programmable digital neuromorphic architecture with high-density 3d memory
Duckhwan Kim, Jaeha Kung, Sek Chai, Sudhakar Yalamanchili, and Saibal Mukhopadhyay · 2016
Earlier work this paper cites.
Overview of 3-d architecture design opportunities and techniques
Jishen Zhao, Qiaosha Zou, and Yuan Xie · 2017
Earlier work this paper cites.
A discourse-aware attention model for abstractive summarization of long documents
Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian · 2018
Earlier work this paper cites.
Language models are unsupervised multitask learners
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al · 2019
Earlier work this paper cites.
Fastertransformer
NVIDIA Corporation · 2019
Earlier work this paper cites.
Retransformer: Reram-based processing-in-memory architecture for transformer acceleration
Xiaoxuan Yang, Bonan Yan, Hai Li, and Yiran Chen · 2020
Earlier work this paper cites.
Evaluating large language models trained on code
Original
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al · 2021
Earlier work this paper cites.
Terapipe: Token-level pipeline parallelism for training large-scale language models
Zhuohan Li, Siyuan Zhuang, Shiyuan Guo, Danyang Zhuo, Hao Zhang, Dawn Song, and Ion Stoica · 2021
Earlier work this paper cites.
In-memory computing based accelerator for transformer networks for long sequences
Ann Franchesca Laguna, Arman Kazemi, Michael Niemier, and X. Sharon Hu · 2021
Earlier work this paper cites.
Introducing chatgpt
OpenAI · 2022
Earlier work this paper cites.
Nvidia h100 tensor core gpu architecture
NVIDIA · 2022
Earlier work this paper cites.
Orca: A distributed serving system for { \{ Transformer-Based } \} generative models
Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun · 2022
Earlier work this paper cites.
Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale
Reza Yazdani Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Minjia Zhang, Jeff Rasley, et al · 2022
Earlier work this paper cites.
Llama 2: Open foundation and fine-tuned chat models, 2023
Original
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al · 2023
Earlier work this paper cites.
Memgpt: Towards llms as operating systems
Original
Charles Packer, Vivian Fang, Shishir G Patil, Kevin Lin, Sarah Wooders, and Joseph E Gonzalez · 2023
Earlier work this paper cites.
Splitwise: Efficient generative llm inference using phase splitting
Original
Pratyush Patel, Esha Choukse, Chaojie Zhang, Íñigo Goiri, Aashaka Shah, Saeed Maleki, and Ricardo Bianchini · 2023
Earlier work this paper cites.
Llama: Open and efficient foundation language models, 2023
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample · 2023
Earlier work this paper cites.
Efficient memory management for large language model serving with pagedattention
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica · 2023
Earlier work this paper cites.
Deepspeed ulysses: System optimizations for enabling training of extreme long sequence transformer models, 2023
Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Shuaiwen Leon Song, Samyam Rajbhandari, and Yuxiong He · 2023
Earlier work this paper cites.
Ring attention with blockwise transformers for near-infinite context
Hao Liu, Matei Zaharia, and Pieter Abbeel · 2023
Earlier work this paper cites.
Striped attention: Faster ring attention for causal transformers, 2023
William Brandon, Aniruddha Nrusimha, Kevin Qian, Zachary Ankner, Tian Jin, Zhiye Song, and Jonathan Ragan-Kelley · 2023
Earlier work this paper cites.
Lightseq:: Sequence level parallelism for distributed training of long context transformers
Dacheng Li, Rulin Shao, Anze Xie, Eric P Xing, Joseph E Gonzalez, Ion Stoica, Xuezhe Ma, and Hao Zhang · 2023
Earlier work this paper cites.

Similar

Reducing activation recomputation in large transformer models
Vijay Anand Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, and Bryan Catanzaro · 2023
Cited alongside, same era.
Sequence parallelism: Long sequence training from system perspective
Shenggui Li, Fuzhao Xue, Chaitanya Baranwal, Yongbin Li, and Yang You · 2023
Cited alongside, same era.
L-eval: Instituting standardized evaluation for long context language models, 2023
Chenxin An, Shansan Gong, Ming Zhong, Xingjian Zhao, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu · 2023
Cited alongside, same era.
Tensorrt-llm
NVIDIA Corporation · 2023
Cited alongside, same era.
Flexgen: High-throughput generative inference of large language models with a single gpu
Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang · 2023
Cited alongside, same era.
Fast distributed inference serving for large language models
Original
Bingyang Wu, Yinmin Zhong, Zili Zhang, Gang Huang, Xuanzhe Liu, and Xin Jin · 2023
Cited alongside, same era.
Prompt cache: Modular attention reuse for low-latency inference
Original
In Gim, Guojun Chen, Seung-seob Lee, Nikhil Sarda, Anurag Khandelwal, and Lin Zhong · 2023
Cited alongside, same era.
Efficiently programming large language models using sglang
Original
Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Jeff Huang, Chuyue Sun, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al · 2023
Cited alongside, same era.
H3datten: Heterogeneous 3-d integrated hybrid analog and digital compute-in-memory accelerator for vision transformer self-attention
Wantong Li, Madison Manley, James Read, Ankit Kaul, Muhannad S. Bakir, and Shimeng Yu · 2023
Cited alongside, same era.
X-former: In-memory acceleration of transformers
Shrihari Sridharan, Jacob R. Stevens, Kaushik Roy, and Anand Raghunathan · 2023
Cited alongside, same era.
Samsung to unveil 3D DRAM in 2025 to lead AI chip market
The Korea Economic Daily · 2023
Cited alongside, same era.
Kivi : Plug-and-play 2bit kv cache quantization with streaming asymmetric quantization
Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu · 2023
Cited alongside, same era.

Then

Attentionstore: Cost-effective attention reuse across multi-turn conversations in large language model serving
Original
Bin Gao, Zhuomin He, Puru Sharma, Qingxuan Kang, Djordje Jevdjic, Junbo Deng, Xingkun Yang, Zhou Yu, and Pengfei Zuo · 2024
Closest in time.
Preble: Efficient distributed prompt scheduling for llm serving
Vikranth Srivatsa, Zijian He, Reyna Abhyankar, Dongming Li, and Yiying Zhang · 2024
Closest in time.
H3dm: A high-bandwidth high-capacity hybrid 3d memory design for gpus
Negar Akbarzadeh, Sina Darabi, Atiyeh Gheibi-Fetrat, Amir Mirzaei, Mohammad Sadrosadati, and Hamid Sarbazi-Azad · 2024
Closest in time.
Efficient and economic large language model inference with attention offloading, 2024
Shaoyuan Chen, Yutong Lin, Mingxing Zhang, and Yongwei Wu · 2024
Closest in time.
Effectively compress kv heads for llm, 2024
Hao Yu, Zelan Yang, Shen Li, Yong Li, and Jianxin Wu · 2024
Closest in time.
Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling, 2024
Zefan Cai., Yichi Zhang, Bofei Gao, Yuliang Liu, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, and Wen Xiao · 2024
Closest in time.
Zipcache: Accurate and efficient kv cache quantization with salient token identification, 2024
Yefei He, Luoming Zhang, Weijia Wu, Jing Liu, Hong Zhou, and Bohan Zhuang · 2024
Closest in time.
Intactkv: Improving large language model quantization by keeping pivot tokens intact, 2024
Ruikang Liu, Haoli Bai, Haokun Lin, Yuening Li, Han Gao, Zhengzhuo Xu, Lu Hou, Jun Yao, and Chun Yuan · 2024
Closest in time.
Minicache: Kv cache compression in depth dimension for large language models, 2024
Akide Liu, Jing Liu, Zizheng Pan, Yefei He, Gholamreza Haffari, and Bohan Zhuang · 2024
Closest in time.
Attention score is not all you need for token importance indicator in kv cache reduction: Value also matters, 2024
Zhiyu Guo, Hidetaka Kamigaito, and Taro Watanabe · 2024
Closest in time.
A simple and effective l 2 l_{2} norm-based strategy for kv cache compression, 2024
Alessio Devoto, Yu Zhao, Simone Scardapane, and Pasquale Minervini · 2024
Closest in time.
Sirllm: Streaming infinite retentive llm, 2024
Yao Yao, Zuchao Li, and Hai Zhao · 2024
Closest in time.
Pyramidinfer: Pyramid kv cache compression for high-throughput llm inference, 2024
Dongjie Yang, XiaoDong Han, Yan Gao, Yao Hu, Shilin Zhang, and Hai Zhao · 2024
Closest in time.
Keyformer: Kv cache reduction through key tokens selection for efficient generative inference, 2024
Muhammad Adnan, Akhil Arunkumar, Gaurav Jain, Prashant J. Nair, Ilya Soloveychik, and Purushotham Kamath · 2024
Closest in time.
Snapkv: Llm knows what you are looking for before generation, 2024
Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen · 2024
Closest in time.
Mlkv: Multi-layer key-value heads for memory efficient transformer decoding, 2024
Zayd Muhammad Kawakibi Zuhri, Muhammad Farid Adilazuarda, Ayu Purwarianti, and Alham Fikri Aji · 2024
Closest in time.
Layer-condensed kv cache for efficient inference of large language models, 2024
Haoyi Wu and Kewei Tu · 2024
Closest in time.
You only cache once: Decoder-decoder architectures for language models, 2024
Yutao Sun, Li Dong, Yi Zhu, Shaohan Huang, Wenhui Wang, Shuming Ma, Quanlu Zhang, Jianyong Wang, and Furu Wei · 2024
Closest in time.
Mamba: Linear-time sequence modeling with selective state spaces, 2024
Albert Gu and Tri Dao · 2024
Closest in time.
Transformers are ssms: Generalized models and efficient algorithms through structured state space duality, 2024
Tri Dao and Albert Gu · 2024
Closest in time.
Jamba: A hybrid transformer-mamba language model, 2024
Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, Omri Abend, Raz Alon, Tomer Asida, Amir Bergman, Roman Glozman, Michael Gokhman, Avashalom Manevich, Nir Ratner, Noam Rozen, Erez Shwartz, Mor Zusman, and Yoav Shoham · 2024
Closest in time.
Recurrentgemma: Moving past transformers for efficient open language models, 2024
Aleksandar Botev, Soham De, Samuel L Smith, Anushan Fernando, George-Cristian Muraru, Ruba Haroun, Leonard Berrada, Razvan Pascanu, Pier Giuseppe Sessa, Robert Dadashi, Léonard Hussenot, Johan Ferret, Sertan Girgin, Olivier Bachem, Alek Andreev, Kathleen Kenealy, Thomas Mesnard, Cassidy Hardin, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Armand Joulin, Noah Fiedel, Evan Senter, Yutian Chen, Srivatsan Srinivasan, Guillaume Desjardins, David Budden, Arnaud Doucet, Sharad Vikram, Adam Paszke, Trevor Gale, Sebastian Borgeaud, Charlie Chen, Andy Brock, Antonia Paterson, Jenny Brennan, Meg Risdal, Raj Gundluru, Nesh Devanathan, Paul Mooney, Nilay Chauhan, Phil Culliton, Luiz GUStavo Martins, Elisa Bandy, David Huntsperger, Glenn Cameron, Arthur Zucker, Tris Warkentin, Ludovic Peran, Minh Giang, Zoubin Ghahramani, Clément Farabet, Koray Kavukcuoglu, Demis Hassabis, Raia Hadsell, Yee Whye Teh, and Nando de Frietas · 2024
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…