SPIN: Accelerating Large Language Model Inference with Heterogeneous Speculative Models

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems , vol. 30, 2017

2017

Earlier work this paper cites.

T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al. , “Language models are few-shot learners,” Advances in neural information processing systems , vol. 33, pp. 1877–1901, 2020

2020

Earlier work this paper cites.

J. Fang, Y. Yu, C. Zhao, and J. Zhou, “Turbotransformers: an efficient gpu serving system for transformer models,” in Proceedings of the 26th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming , 2021, pp. 389–402

2021

Earlier work this paper cites.

B. Fu, F. Chen, P. Li, and D. Zeng, “Tcb: Accelerating transformer inference services with request concatenation,” in Proceedings of the 51st International Conference on Parallel Processing , 2022, pp. 1–11

2022

Earlier work this paper cites.

G.-I. Yu, J. S. Jeong, G.-W. Kim, S. Kim, and B.-G. Chun, “Orca: A distributed serving system for transformer–based generative models,” in 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22) , 2022, pp. 521–538

2022

Earlier work this paper cites.

J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023

Original

2023

Earlier work this paper cites.

W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong et al. , “A survey of large language models,” arXiv preprint arXiv:2303.18223 , 2023

Original

2023

Earlier work this paper cites.

Y. Wang, K. Chen, H. Tan, and K. Guo, “Tabi: An efficient multi-level inference system for large language models,” in Proceedings of the Eighteenth European Conference on Computer Systems , 2023, pp. 233–248

2023

Earlier work this paper cites.

Z. Liu, J. Wang, T. Dao, T. Zhou, B. Yuan, Z. Song, A. Shrivastava, C. Zhang, Y. Tian, C. Re et al. , “Deja vu: Contextual sparsity for efficient llms at inference time,” in International Conference on Machine Learning . PMLR, 2023, pp. 22 137–22 176

2023

Earlier work this paper cites.

C. Chen, S. Borgeaud, G. Irving, J.-B. Lespiau, L. Sifre, and J. Jumper, “Accelerating large language model decoding with speculative sampling,” arXiv preprint arXiv:2302.01318 , 2023

Original

2023

Earlier work this paper cites.

Y. Leviathan, M. Kalman, and Y. Matias, “Fast inference from transformers via speculative decoding,” in International Conference on Machine Learning . PMLR, 2023, pp. 19 274–19 286

2023

Earlier work this paper cites.

Y. Zhai, C. Jiang, L. Wang, X. Jia, S. Zhang, Z. Chen, X. Liu, and Y. Zhu, “Bytetransformer: A high-performance transformer boosted for variable-length inputs,” in 2023 IEEE International Parallel and Distributed Processing Symposium (IPDPS) . IEEE, 2023, pp. 344–355

2023

Earlier work this paper cites.

W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with pagedattention,” in Proceedings of the 29th Symposium on Operating Systems Principles , 2023, pp. 611–626

2023

Earlier work this paper cites.

B. Peng, C. Li, P. He, M. Galley, and J. Gao, “Instruction tuning with gpt-4,” arXiv preprint arXiv:2304.03277 , 2023

Original

2023

Earlier work this paper cites.

M. Rashad, “Chatgpt-prompts,” 2023. [Online]. Available: https://huggingface.co/datasets/MohamedRashad/ChatGPT-prompts

2023

Cited alongside, same era.

A. Palla, “Chatbot instruction prompts,” 2023. [Online]. Available: https://huggingface.co/datasets/alespalla/chatbot_instruction_prompts

2023

Cited alongside, same era.

HuggingFace, “Large language model text generation inference,” 2023. [Online]. Available: https://github.com/huggingface/text-generation-inference

2023

Cited alongside, same era.

Y. Fu, P. Bailis, I. Stoica, and H. Zhang, “Breaking the sequential dependency of llm inference using lookahead decoding,” November 2023. [Online]. Available: https://lmsys.org/blog/2023-11-21-lookahead-decoding/

2023

Cited alongside, same era.

Q. Su, C. Giannoula, and G. Pekhimenko, “The synergy of speculative decoding and batching in serving large language models,” arXiv preprint arXiv:2310.18813 , 2023

Y. Li, F. Wei, C. Zhang, and H. Zhang, “Eagle: Speculative sampling requires rethinking feature uncertainty,” arXiv preprint arXiv:2401.15077 , 2024

Original

2024

Later among the works it cites.

M. Elhoushi, A. Shrivastava, D. Liskovich, B. Hosmer, B. Wasti, L. Lai, A. Mahmoud, B. Acun, S. Agarwal, A. Roman et al. , “Layer skip: Enabling early exit inference and self-speculative decoding,” arXiv preprint arXiv:2404.16710 , 2024

Original

2024

Later among the works it cites.

J. Liu, Q. Wang, J. Wang, and X. Cai, “Speculative decoding via early-exiting for faster llm inference with thompson sampling control mechanism,” arXiv preprint arXiv:2406.03853 , 2024

Original

2024

Later among the works it cites.

B. Sun, Z. Huang, H. Zhao, W. Xiao, X. Zhang, Y. Li, and W. Lin, “Llumnix: Dynamic scheduling for large language model serving,” in 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24) . USENIX Association, 2024, pp. 173–191

2024

Later among the works it cites.