ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compression

J. W. Rae, A. Potapenko, S. M. Jayakumar, and T. P. Lillicrap, “Compressive transformers for long-range sequence modelling,” 2019. [Online]. Available: https://arxiv.org/abs/1911.05507

Original

1911

Earlier work this paper cites.

S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. S. Koura, A. Sridhar, T. Wang, and L. Zettlemoyer, “Opt: Open pre-trained transformer language models,” 2022. [Online]. Available: https://arxiv.org/abs/2205.01068

Original

2022

Earlier work this paper cites.

Y. Bai, X. Lv, J. Zhang, H. Lyu, J. Tang, Z. Huang, Z. Du, X. Liu, A. Zeng, L. Hou, Y. Dong, J. Tang, and J. Li, “Longbench: A bilingual, multitask benchmark for long context understanding,” arXiv preprint arXiv:2308.14508 , 2023

Original

2023

Earlier work this paper cites.

Meta, “Llama 2: Open foundation and fine-tuned chat models,” 2023. [Online]. Available: https://arxiv.org/abs/2307.09288

Original

2023

Earlier work this paper cites.

D. Li, R. Shao, A. Xie, Y. Sheng, L. Zheng, E. Gonzalez, I. Stoica, X. Ma, and H. Zhang, “How long can open-source llms truly promise on context length?” June 2023. [Online]. Available: https://lmsys.org/blog/2023-06-29-longchat

2023

Earlier work this paper cites.

Z. Zhang, Y. Sheng, T. Zhou, T. Chen, L. Zheng, R. Cai, Z. Song, Y. Tian, C. Ré, C. Barrett, Z. A. Wang, and B. Chen, “H2o: Heavy-hitter oracle for efficient generative inference of large language models,” in Advances in Neural Information Processing Systems , A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, Eds., vol. 36. Curran Associates, Inc., 2023, pp. 34 661–34 710

2023

Earlier work this paper cites.

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” 2023. [Online]. Available: https://arxiv.org/abs/1706.03762

Original

2023

Earlier work this paper cites.

A. M. Ikotun, A. E. Ezugwu, L. Abualigah, B. Abuhaija, and J. Heming, “K-means clustering algorithms: A comprehensive review, variants analysis, and advances in the era of big data,” Information Sciences , vol. 622, pp. 178–210, 2023. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0020025522014633

2023

Earlier work this paper cites.

Y. Sheng, L. Zheng, B. Yuan, Z. Li, M. Ryabinin, D. Y. Fu, Z. Xie, B. Chen, C. Barrett, J. E. Gonzalez, P. Liang, C. Ré, I. Stoica, and C. Zhang, “Flexgen: High-throughput generative inference of large language models with a single gpu,” 2023. [Online]. Available: https://arxiv.org/abs/2303.06865

Original

2023

Cited alongside, same era.

C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan, “SWE-bench: Can language models resolve real-world github issues?” in The Twelfth International Conference on Learning Representations , 2024. [Online]. Available: https://openreview.net/forum?id=VTF8yNQM66

2024

Cited alongside, same era.

OpenAI, “Openai o1 system card,” 2024. [Online]. Available: https://cdn.openai.com/o1-system-card-20240917.pdf

2024

Cited alongside, same era.

OpenAI, “Gpt-4o system card,” 2024. [Online]. Available: https://cdn.openai.com/gpt-4o-system-card.pdf

2024

Cited alongside, same era.

M. Adnan, A. Arunkumar, G. Jain, P. J. Nair, I. Soloveychik, and P. Kamath, “Keyformer: Kv cache reduction through key tokens selection for efficient generative inference,” MLSys , 2024

2024

Closest in time.

Y. Zhao, D. Wu, and J. Wang, “Alisa: Accelerating large language model inference via sparsity-aware kv caching,” 2024. [Online]. Available: https://arxiv.org/abs/2403.17312

Original

2024

Closest in time.

Z. Ning, J. Zhao, Q. Jin, W. Ding, and M. Guo, “Inf-mllm: Efficient streaming inference of multimodal large language models on a single gpu,” 2024. [Online]. Available: https://arxiv.org/abs/2409.09086

Original

2024

Closest in time.

J. Tang, Y. Zhao, K. Zhu, G. Xiao, B. Kasikci, and S. Han, “Quest: Query-aware sparsity for efficient long-context llm inference,” ICML , 2024

2024

Closest in time.

K. Ataallah, C. Gou, E. Abdelrahman, K. Pahwa, J. Ding, and M. Elhoseiny, “Infinibench: A comprehensive benchmark for large multimodal models in very long video understanding,” 2024. [Online]. Available: https://arxiv.org/abs/2406.19875