S2D: Sorted Speculative Decoding For More Efficient Deployment of Nested Large Language Models

Language models are few-shot learners

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020 · 1901

Earlier work this paper cites.

Blockwise parallel decoding for deep autoregressive models

Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. 2018 · 2018

Earlier work this paper cites.

Evaluating large language models trained on code

Original

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021 · 2021

Earlier work this paper cites.

Efficient large-scale language model training on gpu clusters using megatron-lm

Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, et al. 2021 · 2021

Earlier work this paper cites.

Instantaneous grammatical error correction with shallow aggressive decoding

Original

Xin Sun, Tao Ge, Furu Wei, and Houfeng Wang. 2021 · 2021

Earlier work this paper cites.

Lossless speedup of autoregressive translation with generalized aggressive decoding

Original

Heming Xia, Tao Ge, Furu Wei, and Zhifang Sui. 2022 · 2022

Earlier work this paper cites.

Jointly-learned exit and inference for a dynamic neural network

Joud Chataoui, Mark Coates, et al. 2023 · 2023

Earlier work this paper cites.

Rest: Retrieval-based speculative decoding

Original

Zhenyu He, Zexuan Zhong, Tianle Cai, Jason D Lee, and Di He. 2023 · 2023

Earlier work this paper cites.

Online speculative decoding

Original

Xiaoxuan Liu, Lanxiang Hu, Peter Bailis, Ion Stoica, Zhijie Deng, Alvin Cheung, and Hao Zhang. 2023 · 2023

Earlier work this paper cites.

Specinfer: Accelerating generative llm serving with speculative inference and token tree verification

Original

Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Rae Ying Yee Wong, Zhuoming Chen, Daiyaan Arfeen, Reyna Abhyankar, and Zhihao Jia. 2023 · 2023

Earlier work this paper cites.

Spectr: Fast speculative decoding via optimal transport

Ziteng Sun, Ananda Theertha Suresh, Jae Hun Ro, Ahmad Beirami, Himanshu Jain, and Felix Yu. 2023 · 2023

Cited alongside, same era.

Alpaca: A strong, replicable instruction-following model

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023 · 2023

Cited alongside, same era.

Gemini: a family of highly capable multimodal models

Original

Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023 · 2023

Cited alongside, same era.

Llama 2: Open foundation and fine-tuned chat models

Original

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023 · 2023

Cited alongside, same era.

Sortednet, a place for every network and every network in its place: Towards a generalized solution for training many-in-one neural networks

Layer skip: Enabling early exit inference and self-speculative decoding

Original

Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, et al. 2024 · 2024

Closest in time.

Break the sequential dependency of llm inference using lookahead decoding

Original

Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. 2024 · 2024

Closest in time.

Sorted llama: Unlocking the potential of intermediate layers of large language models for dynamic inference

Parsa Kavehzadeh, Mojtaba Valipour, Marzieh Tahaei, Ali Ghodsi, Boxing Chen, and Mehdi Rezagholizadeh. 2024 · 2024

Closest in time.

Eagle: Speculative sampling requires rethinking feature uncertainty

Original

Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2024 · 2024

Closest in time.

Panda: Preference adaptation for enhancing domain-specific abilities of llms