AMUSD: Asynchronous Multi-Device Speculative Decoding for LLM Acceleration

A. Vaswani, “Attention is all you need,” Advances in Neural Information Processing Systems , 2017

2017

Earlier work this paper cites.

M. Stern, N. Shazeer, and J. Uszkoreit, “Blockwise parallel decoding for deep autoregressive models,” Advances in Neural Information Processing Systems , vol. 31, 2018

2018

Earlier work this paper cites.

M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman et al. , “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374 , 2021

Original

2021

Earlier work this paper cites.

J. Zhang, J. Wang, H. Li, L. Shou, K. Chen, G. Chen, and S. Mehrotra, “Draft & verify: Lossless large language model acceleration via self-speculative decoding,” arXiv preprint arXiv:2309.08168 , 2023

Original

2023

Earlier work this paper cites.

X. Hou et al. , “A survey of large language models for code: Evolution, benchmarking, and future trends,” arXiv preprint arXiv:2304.12110 , 2023

Original

2023

Earlier work this paper cites.

Q. Zhou et al. , “Large language models for software engineering: Survey and open problems,” arXiv preprint arXiv:2310.03533 , 2023

Original

2023

Earlier work this paper cites.

B. Roziere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y. Adi, J. Liu, T. Remez, J. Rapin et al. , “Code llama: Open foundation models for code,” arXiv preprint arXiv:2308.12950 , 2023

Original

2023

Cited alongside, same era.

X. Jiang, Y. Dong, L. Wang, F. Zheng, Q. Shang, G. Li, Z. Jin, and W. Jiao, “Self-planning code generation with large language models,” ACM Transactions on Software Engineering and Methodology , 2023

2023

Cited alongside, same era.

L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing et al. , “Judging llm-as-a-judge with mt-bench and chatbot arena,” Advances in Neural Information Processing Systems , vol. 36, pp. 46 595–46 623, 2023

2023

Cited alongside, same era.

H. Xia, Z. Yang, Q. Dong, P. Wang, Y. Li, T. Ge, T. Liu, W. Li, and Z. Sui, “Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding,” arXiv preprint arXiv:2401.07851 , 2024

Original

2024

Cited alongside, same era.

Y. Li, F. Wei, C. Zhang, and H. Zhang, “Eagle: Speculative sampling requires rethinking feature uncertainty,” arXiv preprint arXiv:2401.15077 , 2024

Original

2024

Closest in time.

Z. He, Z. Zhong, T. Cai, J. Lee, and D. He, “Rest: Retrieval-based speculative decoding,” in Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , 2024, pp. 1582–1595

2024

Closest in time.

Y. Zhou, K. Lyu, A. S. Rawat, A. K. Menon, A. Rostamizadeh, S. Kumar, J.-F. Kagy, and R. Agarwal, “Distillspec: Improving speculative decoding via knowledge distillation,” in The Twelfth International Conference on Learning Representations , 2024

2024

Closest in time.

Y. Fu, P. Bailis, I. Stoica, and H. Zhang, “Break the sequential dependency of llm inference using lookahead decoding,” arXiv preprint arXiv:2402.02057 , 2024

Original

2024