Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption

Fast transformer decoding: One write-head is all you need

Original

Noam Shazeer · 1911

Earlier work this paper cites.

Chatgpt and open-ai models: A preliminary review

Konstantinos I. Roumeliotis and Nikolaos D. Tselikas · 1999

Earlier work this paper cites.

Linformer: Self-attention with linear complexity

Original

Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma · 2006

Earlier work this paper cites.

Longformer: The long-document transformer, 2020

Iz Beltagy, Matthew E. Peters, and Arman Cohan · 2020

Earlier work this paper cites.

Language models are few-shot learners

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei · 2020

Earlier work this paper cites.

Gpt-3: Its nature, scope, limits, and consequences

Luciano Floridi and Massimo Chiriatti · 2020

Earlier work this paper cites.

Transformers are RNNs: Fast autoregressive transformers with linear attention

Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret · 2020

Earlier work this paper cites.

Revisiting linformer with a modified self-attention with linear complexity

Original

Madhusudan Verma · 2021

Earlier work this paper cites.

Orca: A distributed serving system for Transformer-Based generative models

Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun · 2022

Earlier work this paper cites.

GQA: Training generalized multi-query transformer models from multi-head checkpoints

Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai · 2023

Earlier work this paper cites.

L-eval: Instituting standardized evaluation for long context language models

Original

Chenxin An, Shansan Gong, Ming Zhong, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu · 2023

Earlier work this paper cites.

Longbench: A bilingual, multitask benchmark for long context understanding

Original

Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li · 2023

Earlier work this paper cites.

Adapting language models to compress contexts

Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen · 2023

Earlier work this paper cites.

Model tells you what to discard: Adaptive KV cache compression for llms

Original

Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao · 2023

Earlier work this paper cites.

LLMLingua: Compressing prompts for accelerated inference of large language models

Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu · 2023

Earlier work this paper cites.

Efficient memory management for large language model serving with pagedattention

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica · 2023

Earlier work this paper cites.

Loogle: Can long-context language models understand long contexts?

Original

Jiaqi Li, Mengmeng Wang, Zilong Zheng, and Muhan Zhang · 2023

Earlier work this paper cites.

Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time

Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava · 2023

Earlier work this paper cites.

Landmark attention: Random-access infinite context length for transformers

Original

Amirkeivan Mohtashami and Martin Jaggi · 2023

Earlier work this paper cites.

Learning to compress prompts with gist tokens

Jesse Mu, Xiang Li, and Noah D. Goodman · 2023

Earlier work this paper cites.

RWKV: reinventing rnns for the transformer era

Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Leon Derczynski, Xingjian Du, Matteo Grella, Kranthi Kiran GV, Xuzheng He, Haowen Hou, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartlomiej Koptyra, Hayden Lau, Jiaju Lin, Krishna Sri Ipsit Mantri, Ferdinand Mom, Atsushi Saito, Guangyu Song, Xiangru Tang, Johan S. Wind, Stanislaw Wozniak, Zhenyuan Zhang, Qinghua Zhou, Jian Zhu, and Rui-Jie Zhu · 2023

Earlier work this paper cites.

Sparq attention: Bandwidth-efficient LLM inference

Original

Luka Ribar, Ivan Chelombiev, Luke Hudlass-Galley, Charlie Blake, Carlo Luschi, and Douglas Orr · 2023

Earlier work this paper cites.

ZeroSCROLLS: A zero-shot benchmark for long text understanding

Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, and Omer Levy · 2023

Cited alongside, same era.

Omniquant: Omnidirectionally calibrated quantization for large language models

Original

Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo · 2023

Cited alongside, same era.

Flexgen: High-throughput generative inference of large language models with a single GPU

Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang · 2023

Cited alongside, same era.

A brief overview of chatgpt: The history, status quo and potential future development

Tianyu Wu, Shizhu He, Jingping Liu, Siqi Sun, Kang Liu, Qing-Long Han, and Yang Tang · 2023

Cited alongside, same era.

Efficient streaming language models with attention sinks, 2023

Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis · 2023

Cited alongside, same era.

LLM maybe longlm: Self-extend LLM context window without tuning

Original

Hongye Jin, Xiaotian Han, Jingfeng Yang, Zhimeng Jiang, Zirui Liu, Chia-Yuan Chang, Huiyuan Chen, and Xia Hu · 2024

Closest in time.

Gear: An efficient kv cache compression recipefor near-lossless generative inference of llm

Original

Hao Kang, Qingru Zhang, Souvik Kundu, Geonhwa Jeong, Zaoxing Liu, Tushar Krishna, and Tuo Zhao · 2024

Closest in time.

In search of needles in a 11m haystack: Recurrent memory finds what llms miss

Original

Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Dmitry Sorokin, Artyom Y. Sorokin, and Mikhail Burtsev · 2024

Closest in time.

Infinigen: Efficient generative inference of large language models with dynamic KV cache management

Wonbeom Lee, Jungi Lee, Junghwan Seo, and Jaewoong Sim · 2024

Closest in time.

Infinite-llm: Efficient LLM service for long context with distattention and distributed kvcache