Decoding Speculative Decoding

Language models are few-shot learners

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020 · 1901

Earlier work this paper cites.

Hellaswag: Can a machine really finish your sentence?

Original

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019 · 1905

Earlier work this paper cites.

Deep encoder, shallow decoder: Reevaluating non-autoregressive machine translation

Original

Jungo Kasai, Nikolaos Pappas, Hao Peng, James Cross, and Noah A Smith. 2020 · 2006

Earlier work this paper cites.

Measuring massive multitask language understanding

Original

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020 · 2009

Earlier work this paper cites.

Lightseq: A high performance inference library for transformers

Original

Xiaohui Wang, Ying Xiong, Yang Wei, Mingxuan Wang, and Lei Li. 2020 · 2010

Earlier work this paper cites.

Computer architecture: a quantitative approach

John L Hennessy and David A Patterson. 2011 · 2011

Earlier work this paper cites.

Blockwise parallel decoding for deep autoregressive models

Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. 2018 · 2018

Earlier work this paper cites.

The design and operation of CloudLab

Dmitry Duplyakin, Robert Ricci, Aleksander Maricq, Gary Wong, Jonathon Duerig, Eric Eide, Leigh Stoller, Mike Hibler, David Johnson, Kirk Webb, Aditya Akella, Kuangching Wang, Glenn Ricart, Larry Landweber, Chip Elliott, Michael Zink, Emmanuel Cecchet, Snigdhaswin Kar, and Prabodh Mishra. 2019 · 2019

Earlier work this paper cites.

Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale

Reza Yazdani Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Minjia Zhang, Jeff Rasley, et al. 2022 · 2022

Earlier work this paper cites.

Confident adaptive language modeling

Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Tran, Yi Tay, and Donald Metzler. 2022 · 2022

Earlier work this paper cites.

Orca: A distributed serving system for { \{ Transformer-Based } \} generative models

Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022 · 2022

Earlier work this paper cites.

Opt: Open pre-trained transformer language models

Original

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022 · 2022

Cited alongside, same era.

Fast and robust early-exiting framework for autoregressive language models with synchronized parallel decoding

Original

Sangmin Bae, Jongwoo Ko, Hwanjun Song, and Se-Young Yun. 2023 · 2023

Cited alongside, same era.

Accelerating large language model decoding with speculative sampling

Original

Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. 2023 · 2023

Cited alongside, same era.

Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023 · 2023

Cited alongside, same era.

Predictive pipelined decoding: A compute-latency trade-off for exact llm decoding

Original

Seongjun Yang, Gibbeum Lee, Jaewoong Cho, Dimitris Papailiopoulos, and Kangwook Lee. 2023 · 2023

Later among the works it cites.

Draft & verify: Lossless large language model acceleration via self-speculative decoding

Original

Jun Zhang, Jue Wang, Huan Li, Lidan Shou, Ke Chen, Gang Chen, and Sharad Mehrotra. 2023 · 2023

Later among the works it cites.

Judging llm-as-a-judge with mt-bench and chatbot arena

Original

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023 · 2023

Later among the works it cites.

Medusa: Simple llm inference acceleration framework with multiple decoding heads

Original

Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao. 2024 · 2024

Closest in time.