vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention

Generating Long Sequences with Sparse Transformers

Original

Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019 · 1904

Earlier work this paper cites.

Fast Transformer Decoding: One Write-Head is All You Need

Original

Noam Shazeer. 2019 · 1911

Earlier work this paper cites.

Longformer: The Long-Document Transformer

Original

Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020 · 2004

Earlier work this paper cites.

Architectural support for address translation on GPUs: designing memory management units for CPU/GPUs with unified address spaces. In Proceedings of the 19th International Conference on Architectural Support for Programming Languages and Operating Systems (Salt Lake City, Utah, USA) (ASPLOS ’14) . Association for Computing Machinery, New York, NY, USA, 743–758

Bharath Pichai, Lisa Hsu, and Abhishek Bhattacharjee. 2014 · 2014

Earlier work this paper cites.

Coordinated and Efficient Huge Page Management with Ingens. In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16) . USENIX Association, Savannah, GA, 705–721

Youngjin Kwon, Hangchen Yu, Simon Peter, Christopher J. Rossbach, and Emmett Witchel. 2016 · 2016

Earlier work this paper cites.

Attention is All you Need. In Advances in Neural Information Processing Systems , I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.), Vol. 30. Curran Associates, Inc

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017 · 2017

Earlier work this paper cites.

Neighborhood-Aware Address Translation for Irregular GPU Applications. In 2018 51st Annual IEEE/ACM International Symposium on Microarchitecture (MICRO) . 352–363

Seunghee Shin, Michael LeBeane, Yan Solihin, and Arkaprava Basu. 2018 · 2018

Earlier work this paper cites.

HawkEye: Efficient Fine-Grained OS Support for Huge Pages. In Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems (Providence, RI, USA) (ASPLOS ’19) . Association for Computing Machinery, New York, NY, USA, 347–360

Ashish Panwar, Sorav Bansal, and K. Gopinath. 2019 · 2019

Earlier work this paper cites.

(Mis)managed: A Novel TLB-based Covert Channel on GPUs. In Proceedings of the 2021 ACM Asia Conference on Computer and Communications Security (Virtual Event, Hong Kong) (ASIA CCS ’21) . Association for Computing Machinery, New York, NY, USA, 872–885

Ajay Nayak, Pratheek B., Vinod Ganapathy, and Arkaprava Basu. 2021 · 2021

Earlier work this paper cites.

FlashAttention

2022 · 2022

Earlier work this paper cites.

PaLM: Scaling Language Modeling with Pathways

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022 · 2022

Earlier work this paper cites.

Designing Virtual Memory System of MCM GPUs. In 2022 55th IEEE/ACM International Symposium on Microarchitecture (MICRO) . 404–422

B Pratheek, Neha Jawalkar, and Arkaprava Basu. 2022 · 2022

Earlier work this paper cites.

Orca: A Distributed Serving System for Transformer-Based Generative Models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22) . USENIX Association, Carlsbad, CA, 521–538

Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022 · 2022

Earlier work this paper cites.

Flash-Decoding for long-context inference

2023 · 2023

Earlier work this paper cites.

FlashInfer: Kernel Library for LLM Serving

2023 · 2023

Earlier work this paper cites.

LightLLM: A Light and Fast Inference Service for LLM

2023 · 2023

Earlier work this paper cites.

Performance decay when using paged attention

2023 · 2023

Earlier work this paper cites.

TensorRT-LLM: A TensorRT Toolbox for Optimized Large Language Model Inference

2023 · 2023

Earlier work this paper cites.

Use optimized kernels for MQA/GQA

2023 · 2023

Earlier work this paper cites.

SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills

Original

Amey Agrawal, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, and Ramachandran Ramjee. 2023 · 2023

Earlier work this paper cites.

GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints

Original

Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. 2023 · 2023

Earlier work this paper cites.

A Case Study in CUDA Kernel Fusion: Implementing FlashAttention-2 on NVIDIA Hopper Architecture using the CUTLASS Library

Original

Ganesh Bikshandi and Jay Shah. 2023 · 2023

Earlier work this paper cites.

FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning

Original

Tri Dao. 2023 · 2023

Earlier work this paper cites.

Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (Koblenz, Germany) (SOSP ’23) . Association for Computing Machinery, New York, NY, USA, 611–626

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023 · 2023

Cited alongside, same era.

SpotServe: Serving Generative Large Language Models on Preemptible Instances

Original

Xupeng Miao, Chunan Shi, Jiangfei Duan, Xiaoli Xi, Dahua Lin, Bin Cui, and Zhihao Jia. 2023 · 2023

Cited alongside, same era.

GPT-4 Technical Report

Original

OpenAI. 2023 · 2023

Cited alongside, same era.

FlexGen: high-throughput generative inference of large language models with a single GPU. In Proceedings of the 40th International Conference on Machine Learning (Honolulu, Hawaii, USA) (ICML’23) . JMLR.org, Article 1288, 23 pages

Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. 2023 · 2023

Mélange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneity

Original

Tyler Griggs, Xiaoxuan Liu, Jiaxiang Yu, Doyoung Kim, Wei-Lin Chiang, Alvin Cheung, and Ion Stoica. 2024 · 2024

Closest in time.

GMLake: Efficient and Transparent GPU Memory Defragmentation for Large-scale DNN Training with Virtual Memory Stitching. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (La Jolla, CA, USA) (ASPLOS ’24) . Association for Computing Machinery, New York, NY, USA, 450–466

Cong Guo, Rui Zhang, Jiale Xu, Jingwen Leng, Zihan Liu, Ziyu Huang, Minyi Guo, Hao Wu, Shouren Zhao, Junping Zhao, and Ke Zhang. 2024 · 2024

Closest in time.

DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inference

Original

Connor Holmes, Masahiro Tanaka, Michael Wyatt, Ammar Ahmad Awan, Jeff Rasley, Samyam Rajbhandari, Reza Yazdani Aminabadi, Heyang Qin, Arash Bakhtiari, Lev Kurilenko, and Yuxiong He. 2024 · 2024

Closest in time.