AdaSpec: Adaptive Speculative Decoding for Fast, SLO-Aware Large Language Model Serving

A neural probabilistic language model

Yoshua Bengio, Réjean Ducharme, and Pascal Vincent. 2000 · 2000

Earlier work this paper cites.

Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond. In Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning , Stefan Riezler and Yoav Goldberg (Eds.). Association for Computational Linguistics, Berlin, Germany, 280–290

Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Çağlar Gu ˙ \dot{} lçehre, and Bing Xiang. 2016 · 2016

Earlier work this paper cites.

Low latency rnn inference with cellular batching. In Proceedings of the Thirteenth EuroSys Conference . 1–15

Pin Gao, Lingfan Yu, Yongwei Wu, and Jinyang Li. 2018 · 2018

Earlier work this paper cites.

Natural Questions: A Benchmark for Question Answering Research

Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019 · 2019

Earlier work this paper cites.

MArk: Exploiting Cloud Services for Cost-Effective, SLO-Aware Machine Learning Inference Serving. In 2019 USENIX Annual Technical Conference (USENIX ATC 19) . USENIX Association, Renton, WA, 1049–1062

Chengliang Zhang, Minchen Yu, Wei Wang, and Feng Yan. 2019 · 2019

Earlier work this paper cites.

Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems , H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 1877–1901

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020 · 2020

Earlier work this paper cites.

Dense Passage Retrieval for Open-Domain Question Answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (Eds.). Association for Computational Linguistics, Online, 6769–6781

Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020 · 2020

Earlier work this paper cites.

Transformers: State-of-the-Art Natural Language Processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations . Association for Computational Linguistics, Online, 38–45

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020 · 2020

Earlier work this paper cites.

Training verifiers to solve math word problems

Original

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al · 2021

Earlier work this paper cites.

Scrooge: A Cost-Effective Deep Learning Inference System. In Proceedings of the ACM Symposium on Cloud Computing (Seattle, WA, USA) (SoCC ’21) . Association for Computing Machinery, New York, NY, USA, 624–638

Yitao Hu, Rajrup Ghosh, and Ramesh Govindan. 2021 · 2021

Earlier work this paper cites.

Orca: A distributed serving system for { \{ Transformer-Based } \} generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22) . 521–538

Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022 · 2022

Earlier work this paper cites.

Gpt-4 technical report

Original

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al · 2023

Earlier work this paper cites.

Accelerating large language model decoding with speculative sampling

Original

Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. 2023 · 2023

Earlier work this paper cites.

Speculative Decoding with Big Little Decoder. In Advances in Neural Information Processing Systems , A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36. Curran Associates, Inc., 39236–39256

Sehoon Kim, Karttikeya Mangalam, Suhong Moon, Jitendra Malik, Michael W Mahoney, Amir Gholami, and Kurt Keutzer. 2023 · 2023

Earlier work this paper cites.

Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (Koblenz, Germany) (SOSP ’23) . Association for Computing Machinery, New York, NY, USA, 611–626

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023 · 2023

Earlier work this paper cites.

Fast Inference from Transformers via Speculative Decoding. In Proceedings of the 40th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 202) , Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (Eds.). PMLR, 19274–19286

Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023 · 2023

Cited alongside, same era.

SpecTr: Fast Speculative Decoding via Optimal Transport. In Advances in Neural Information Processing Systems , A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36. Curran Associates, Inc., 30222–30242

Ziteng Sun, Ananda Theertha Suresh, Jae Hun Ro, Ahmad Beirami, Himanshu Jain, and Felix Yu. 2023 · 2023

Cited alongside, same era.

{ \{ SHEPHERD } \} : Serving { \{ DNNs } \} in the wild. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23) . 787–808

Hong Zhang, Yupeng Tang, Anurag Khandelwal, and Ion Stoica. 2023 · 2023

Cited alongside, same era.

Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track

SSSD: Simply-Scalable Speculative Decoding

Original

Michele Marzollo, Jiawei Zhuang, Niklas Roemer, Lorenz K Müller, and Lukas Cavigelli. 2024 · 2024

Later among the works it cites.

SpecInfer: Accelerating Large Language Model Serving with Tree-based Speculative Inference and Verification. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3 (La Jolla, CA, USA) (ASPLOS ’24) . Association for Computing Machinery, New York, NY, USA, 932–949

Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, Chunan Shi, Zhuoming Chen, Daiyaan Arfeen, Reyna Abhyankar, and Zhihao Jia. 2024 · 2024

Later among the works it cites.

TensorRT-LLM

NVIDIA. 2024 · 2024

Later among the works it cites.

Splitwise: Efficient generative LLM inference using phase splitting. In ISCA

Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. 2024 · 2024

Later among the works it cites.