POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inference

A study of Persistent Threads style GPU programming for GPGPU workloads. In 2012 Innovative Parallel Computing (InPar) . 1–14

Kshitij Gupta, Jeff A. Stuart, and John D. Owens. 2012 · 2012

Earlier work this paper cites.

Kernel Weaver: Automatically Fusing Database Primitives for Efficient GPU Computation. In 2012 45th Annual IEEE/ACM International Symposium on Microarchitecture . 107–118

Haicheng Wu, Gregory Diamos, Srihari Cadambi, and Sudhakar Yalamanchili. 2012 · 2012

Earlier work this paper cites.

Improving GPGPU concurrency with elastic kernels. In Proceedings of the Eighteenth International Conference on Architectural Support for Programming Languages and Operating Systems (Houston, Texas, USA) (ASPLOS ’13) . Association for Computing Machinery, New York, NY, USA, 407–418

Sreepathi Pai, Matthew J. Thazhuthaveetil, and R. Govindarajan. 2013 · 2013

Earlier work this paper cites.

Design and evaluation of the gemtc framework for GPU-enabled many-task computing. In Proceedings of the 23rd International Symposium on High-Performance Parallel and Distributed Computing (Vancouver, BC, Canada) (HPDC ’14) . Association for Computing Machinery, New York, NY, USA, 153–164

Scott J. Krieder, Justin M. Wozniak, Timothy Armstrong, Michael Wilde, Daniel S. Katz, Benjamin Grimmer, Ian T. Foster, and Ioan Raicu. 2014 · 2014

Earlier work this paper cites.

Efficient GPU Spatial-Temporal Multitasking

Yun Liang, Huynh Phung Huynh, Kyle Rupnow, Rick Siow Mong Goh, and Deming Chen. 2015 · 2014

Earlier work this paper cites.

Scalable Kernel Fusion for Memory-Bound GPU Applications. In SC ’14: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis . 191–202

Mohamed Wahib and Naoya Maruyama. 2014 · 2014

Earlier work this paper cites.

Enabling and Exploiting Flexible Task Assignment on GPU through SM-Centric Program Transformations. In Proceedings of the 29th ACM on International Conference on Supercomputing (Newport Beach, California, USA) (ICS ’15) . Association for Computing Machinery, New York, NY, USA, 119–130

Bo Wu, Guoyang Chen, Dong Li, Xipeng Shen, and Jeffrey Vetter. 2015 · 2015

Earlier work this paper cites.

Simultaneous Multikernel GPU: Multi-tasking throughput processors via fine-grained sharing. In 2016 IEEE International Symposium on High Performance Computer Architecture (HPCA) . 358–369

Zhenning Wang, Jun Yang, Rami Melhem, Bruce Childers, Youtao Zhang, and Minyi Guo. 2016 · 2016

Earlier work this paper cites.

Programming Tensor Cores in CUDA 9

Jeremy Appleyard and Scott Yokim. 2017 · 2017

Earlier work this paper cites.

Pagoda: Fine-Grained GPU Resource Virtualization for Narrow Tasks. In Proceedings of the 22nd ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming (Austin, Texas, USA) (PPoPP ’17) . Association for Computing Machinery, New York, NY, USA, 221–234

Tsung Tai Yeh, Amit Sabne, Putt Sakdhnagool, Rudolf Eigenmann, and Timothy G. Rogers. 2017 · 2017

Earlier work this paper cites.

FlashAttention

2022 · 2022

Earlier work this paper cites.

FLASHATTENTION: fast and memory-efficient exact attention with IO-awareness. In Proceedings of the 36th International Conference on Neural Information Processing Systems (New Orleans, LA, USA) (NIPS ’22) . Curran Associates Inc., Red Hook, NY, USA, Article 1189, 16 pages

Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022 · 2022

Earlier work this paper cites.

Breaking the Computation and Communication Abstraction Barrier in Distributed Machine Learning Workloads. In Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (Lausanne, Switzerland) (ASPLOS ’22) . Association for Computing Machinery, New York, NY, USA, 402–416

Abhinav Jangda, Jun Huang, Guodong Liu, Amir Hossein Nodehi Sabet, Saeed Maleki, Youshan Miao, Madanlal Musuvathi, Todd Mytkowicz, and Olli Saarikivi. 2022 · 2022

Earlier work this paper cites.

Automatic Horizontal Fusion for GPU Kernels. In 2022 IEEE/ACM International Symposium on Code Generation and Optimization (CGO) . 14–27

Ao Li, Bojian Zheng, Gennady Pekhimenko, and Fan Long. 2022 · 2022

Earlier work this paper cites.

Orca: A Distributed Serving System for Transformer-Based Generative Models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22) . USENIX Association, Carlsbad, CA, 521–538

Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022 · 2022

Earlier work this paper cites.

ISPA: Exploiting Intra-SM Parallelism in GPUs via Fine-Grained Resource Management

Han Zhao, Weihao Cui, Quan Chen, and Minyi Guo. 2023 · 2022

Earlier work this paper cites.

TensorRT-LLM: A TensorRT Toolbox for Optimized Large Language Model Inference

2023 · 2023

Earlier work this paper cites.

SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills

Original

Amey Agrawal, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, and Ramachandran Ramjee. 2023 · 2023

Earlier work this paper cites.

GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 4895–4901

Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. 2023 · 2023

Cited alongside, same era.

Flash-Decoding for long-context inference

Tri Dao, Daniel Haziza, Francisco Massa, and Grigory Sizov. 2023 · 2023

Cited alongside, same era.

Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (Koblenz, Germany) (SOSP ’23) . Association for Computing Machinery, New York, NY, USA, 611–626

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023 · 2023

Cited alongside, same era.

Stream-K: Work-Centric Parallel Decomposition for Dense Matrix-Matrix Multiplication on the GPU. In Proceedings of the 28th ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming (Montreal, QC, Canada) (PPoPP ’23) . Association for Computing Machinery, New York, NY, USA, 429–431

FlashDecoding++: Faster Large Language Model Inference with Asynchronization, Flat GEMM Optimization, and Heuristics. In Proceedings of Machine Learning and Systems , P. Gibbons, G. Pekhimenko, and C. De Sa (Eds.), Vol. 6. 148–161

Ke Hong, Guohao Dai, Jiaming Xu, Qiuli Mao, Xiuhong Li, Jun Liu, kangdi chen, Yuhan Dong, and Yu Wang. 2024 · 2024

Closest in time.

Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloads

Original

Cunchen Hu, Heyang Huang, Liangliang Xu, Xusheng Chen, Jiang Xu, Shuang Chen, Hao Feng, Chenxi Wang, Sa Wang, Yungang Bao, Ninghui Sun, and Yizhou Shan. 2024 · 2024

Closest in time.

Toward Efficient Inference for Mixture of Experts. In The Thirty-eighth Annual Conference on Neural Information Processing Systems

Haiyang Huang, Newsha Ardalani, Anna Sun, Liu Ke, Shruti Bhosale, Hsien-Hsin S. Lee, Carole-Jean Wu, and Benjamin Lee. 2024 · 2024

Closest in time.

A Framework for Fine-Grained Synchronization of Dependent GPU Kernels. In Proceedings of the 2024 IEEE/ACM International Symposium on Code Generation and Optimization (Edinburgh, United Kingdom) (CGO ’24) . IEEE Press, 93–105