Fetching the paper…
Reading the bibliography…
Key-Value (KV) cache has become a bottleneck of LLMs for long-context generation.
ROUGE: A package for automatic evaluation of summaries
Chin-Yew Lin. 2004 · 2004
Earlier work this paper cites.
Kv-cache: A scalable high-performance web-object cache for manycore
Daniel Waddington, Juan Colmenares, Jilong Kuang, and Fengguang Song. 2013 · 2013
Earlier work this paper cites.
Training verifiers to solve math word problems
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021 · 2021
Earlier work this paper cites.
Measuring massive multitask language understanding
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021 · 2021
Earlier work this paper cites.
Calibrate before use: Improving few-shot performance of language models
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021 · 2021
Earlier work this paper cites.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023 · 2023
Earlier work this paper cites.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023 · 2023
Earlier work this paper cites.
Efficient memory management for large language model serving with pagedattention
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023 · 2023
Earlier work this paper cites.
H2o: Heavy-hitter oracle for efficient generative inference of large language models
Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. 2023 · 2023
Earlier work this paper cites.
The claude 3 model family: Opus, sonnet, haiku
Anthropic. 2024 · 2024
Earlier work this paper cites.
LongBench: A bilingual, multitask benchmark for long context understanding
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024 · 2024
Earlier work this paper cites.
Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling
Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, et al. 2024 · 2024
Earlier work this paper cites.
Magicpig: Lsh sampling for efficient llm generation
Zhuoming Chen, Ranajoy Sadhukhan, Zihao Ye, Yang Zhou, Jianyu Zhang, Niklas Nolte, Yuandong Tian, Matthijs Douze, Leon Bottou, Zhihao Jia, et al. 2024 · 2024
Cited alongside, same era.
Flashattention-2: Faster attention with better parallelism and work partitioning
Tri Dao. 2024 · 2024
Cited alongside, same era.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024 · 2024
Cited alongside, same era.
Ada-kv: Optimizing kv cache eviction by adaptive budget allocation for efficient llm inference
Yuan Feng, Junlin Lv, Yukun Cao, Xike Xie, and S Kevin Zhou. 2024 · 2024
Cited alongside, same era.
Challenges in deploying long-context transformers: A theoretical peak performance analysis
Hierarchical context merging: Better long context understanding for pre-trained LLMs
Woomin Song, Seunghyuk Oh, Sangwoo Mo, Jaehyung Kim, Sukmin Yun, Jung-Woo Ha, and Jinwoo Shin. 2024 · 2024
Closest in time.
QUEST: Query-aware sparsity for efficient long-context LLM inference
Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. 2024 · 2024
Closest in time.
Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. 2024 · 2024
Closest in time.
Seed: Accelerating reasoning tree construction via scheduled speculative decoding
Zhenglin Wang, Jialong Wu, Yilong Lai, Congzhi Zhang, and Deyu Zhou. 2024 · 2024
Closest in time.
Retrieval head mechanistically explains long-context factuality
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Yao Fu. 2024 · 2024
Cited alongside, same era.
Llms need needle in a haystack: Test-pressure testing llms
Greg Kamradt. 2023 · 2024
Cited alongside, same era.
{ \{ InfiniGen } \} : Efficient generative inference of large language models with dynamic { \{ KV } \} cache management
Wonbeom Lee, Jungi Lee, Junghwan Seo, and Jaewoong Sim. 2024 · 2024
Cited alongside, same era.
SnapKV: LLM knows what you are looking for before generation
Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. 2024 · 2024
Cited alongside, same era.
LongGenBench: Long-context generation benchmark
Xiang Liu, Peijie Dong, Xuming Hu, and Xiaowen Chu. 2024c · 2024
Cited alongside, same era.
Keep the cost down: A review on methods to optimize LLM’s KV-cache consumption
Shi Luohe, Hongyi Zhang, Yao Yao, Zuchao Li, and hai zhao. 2024 · 2024
Cited alongside, same era.
Loki: Low-rank keys for efficient sparse attention
Prajwal Singhania, Siddharth Singh, Shwai He, Soheil Feizi, and Abhinav Bhatele. 2024 · 2024
Cited alongside, same era.
Clusterkv: Manipulating llm kv cache in semantic space for recallable compression
Guangda Liu, Chengwei Li, Jieru Zhao, Chenqi Zhang, and Minyi Guo. 2024a
Cited in the paper.
Wenhao Wu, Yizhong Wang, Guangxuan Xiao, Hao Peng, and Yao Fu. 2024 · 2024
Closest in time.
Think: Thinner key cache by query-driven pruning
Yuhui Xu, Zhanming Jie, Hanze Dong, Lei Wang, Xudong Lu, Aojun Zhou, Amrita Saha, Caiming Xiong, and Doyen Sahoo. 2024 · 2024
Closest in time.
PyramidInfer: Pyramid KV cache compression for high-throughput LLM inference
Dongjie Yang, Xiaodong Han, Yan Gao, Yao Hu, Shilin Zhang, and Hai Zhao. 2024b · 2024
Closest in time.
KV cache compression, but what must we give in return? a comprehensive benchmark of long context capable approaches
Jiayi Yuan, Hongyi Liu, Shaochen Zhong, Yu-Neng Chuang, Songchen Li, Guanchu Wang, Duy Le, Hongye Jin, Vipin Chaudhary, Zhaozhuo Xu, Zirui Liu, and Xia Hu. 2024 · 2024
Closest in time.
∞ \infty Bench: Extending long context evaluation beyond 100K tokens
Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Hao, Xu Han, Zhen Thai, Shuo Wang, Zhiyuan Liu, and Maosong Sun. 2024 · 2024
Closest in time.
Atom: Low-bit quantization for efficient and accurate llm serving
Yilong Zhao, Chien-Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen, and Baris Kasikci. 2024 · 2024
Closest in time.
A survey on efficient inference for large language models
Zixuan Zhou, Xuefei Ning, Ke Hong, Tianyu Fu, Jiaming Xu, Shiyao Li, Yuming Lou, Luning Wang, Zhihang Yuan, Xiuhong Li, et al. 2024 · 2024
Closest in time.