Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding

Language models are few-shot learners

Original

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020 · 2005

Earlier work this paper cites.

Computer Architecture: A quantitative approach , 5 edition

John L. Hennessy and David A. Patterson. 2012 · 2012

Earlier work this paper cites.

Blockwise parallel decoding for deep autoregressive models

Original

Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. 2018 · 2018

Earlier work this paper cites.

Fast transformer decoding: One write-head is all you need

Original

Noam Shazeer. 2019 · 2019

Earlier work this paper cites.

The curious case of neural text degeneration

Original

Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020 · 2020

Earlier work this paper cites.

Lamda: Language models for dialog applications

Original

Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Vincent Zhao, Yanqi Zhou, Chung-Ching Chang, Igor Krivokon, Will Rusch, Marc Pickett, Pranesh Srinivasan, Laichee Man, Kathleen Meier-Hellstern, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Arcas, Claire Cui, Marian Croak, Ed Chi, and Quoc Le. 2022 · 2022

Earlier work this paper cites.

Challenges and applications of large language models

Original

Jean Kaddour, Joshua Harris, Maximilian Mozes, Herbie Bradley, Roberta Raileanu, and Robert McHardy. 2023 · 2023

Earlier work this paper cites.

Fast inference from transformers via speculative decoding

Original

Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023 · 2023

Earlier work this paper cites.

Hydra: Sequentially-dependent draft heads for medusa decoding

Original

Zachary Ankner, Rishab Parthasarathy, Aniruddha Nrusimha, Christopher Rinard, Jonathan Ragan-Kelley, and William Brandon. 2024 · 2024

Earlier work this paper cites.

Medusa: Simple llm inference acceleration framework with multiple decoding heads

Original

Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. 2024 · 2024

Earlier work this paper cites.

Layerskip: Enabling early exit inference and self-speculative decoding

Original

Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, Ahmed A Aly, Beidi Chen, and Carole-Jean Wu. 2024 · 2024

Earlier work this paper cites.

Lazyllm: Dynamic token pruning for efficient long context llm inference

Original

Qichen Fu, Minsik Cho, Thomas Merth, Sachin Mehta, Mohammad Rastegari, and Mahyar Najibi. 2024 · 2024

Cited alongside, same era.

Direct alignment of draft model for speculative decoding with chat-fine-tuned llms

Original

Raghavv Goel, Mukul Gagrani, Wonseok Jeon, Junyoung Park, Mingu Lee, and Christopher Lott. 2024 · 2024

Cited alongside, same era.

Graph-structured speculative decoding

Original

Zhuocheng Gong, Jiahao Liu, Ziyue Wang, Pengfei Wu, Jingang Wang, Xunliang Cai, Dongyan Zhao, and Rui Yan. 2024 · 2024

Cited alongside, same era.

Specdec++: Boosting speculative decoding via adaptive candidate lengths

Original

Kaixuan Huang, Xudong Guo, and Mengdi Wang. 2024 · 2024

Cited alongside, same era.

Bass: Batched attention-optimized speculative sampling

Original

Haifeng Qian, Sujan Kumar Gonugondla, Sungsoo Ha, Mingyue Shang, Sanjay Krishna Gouda, Ramesh Nallapati, Sudipta Sengupta, Xiaofei Ma, and Anoop Deoras. 2024 · 2024

Closest in time.

A thorough examination of decoding methods in the era of llms

Original

Chufan Shi, Haoran Yang, Deng Cai, Zhisong Zhang, Yifan Wang, Yujiu Yang, and Wai Lam. 2024 · 2024

Closest in time.

Triforce: Lossless acceleration of long sequence generation with hierarchical speculative decoding

Original

Hanshi Sun, Zhuoming Chen, Xinyu Yang, Yuandong Tian, and Beidi Chen. 2024 · 2024

Closest in time.

Specexec: Massively parallel speculative decoding for interactive llm inference on consumer devices

Original

Ruslan Svirschevski, Avner May, Zhuoming Chen, Beidi Chen, Zhihao Jia, and Max Ryabinin. 2024 · 2024

Closest in time.