PipeSpec: Breaking Stage Dependencies in Hierarchical LLM Decoding

Abstractive text summarization using sequence-to-sequence rnns and beyond

Original

Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. 2016 · 2016

Earlier work this paper cites.

Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization

Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018 · 2018

Earlier work this paper cites.

Blockwise parallel decoding for deep autoregressive models

Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. 2018 · 2018

Earlier work this paper cites.

Evaluating large language models trained on code

Original

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021 · 2021

Earlier work this paper cites.

Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale

Reza Yazdani Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Minjia Zhang, Jeff Rasley, et al. 2022 · 2022

Earlier work this paper cites.

Flashattention: Fast and memory-efficient exact attention with io-awareness

Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022 · 2022

Earlier work this paper cites.

Flashattention-2: Faster attention with better parallelism and work partitioning

Original

Tri Dao. 2023 · 2023

Earlier work this paper cites.

Flashdecoding++: Faster large language model inference on gpus

Original

Ke Hong, Guohao Dai, Jiaming Xu, Qiuli Mao, Xiuhong Li, Jun Liu, Kangdi Chen, Yuhan Dong, and Yu Wang. 2023 · 2023

Earlier work this paper cites.

Efficient memory management for large language model serving with pagedattention

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023 · 2023

Cited alongside, same era.

Fast inference from transformers via speculative decoding

Original

Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023 · 2023

Cited alongside, same era.

Flexgen: High-throughput generative inference of large language models with a single gpu

Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. 2023 · 2023

Cited alongside, same era.

Llama 2: Open foundation and fine-tuned chat models

Original

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023 · 2023

Cited alongside, same era.

Optimizing inference on large language models with nvidia tensorrt-llm, now publicly available

Layer skip: Enabling early exit inference and self-speculative decoding

Original

Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, et al. 2024 · 2024

Later among the works it cites.

Break the sequential dependency of llm inference using lookahead decoding

Original

Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. 2024 · 2024

Later among the works it cites.

Rest: Retrieval-based speculative decoding

Zhenyu He, Zexuan Zhong, Tianle Cai, Jason Lee, and Di He. 2024 · 2024

Later among the works it cites.

Eagle: Speculative sampling requires rethinking feature uncertainty

Original

Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2024 · 2024

Later among the works it cites.

Specinfer: Accelerating large language model serving with tree-based speculative inference and verification