Accelerating LLM Inference with Staged Speculative Decoding

Estimation of probabilities from sparse data for the language model component of a speech recognizer

Katz, S · 1987

Earlier work this paper cites.

Crafting papers on machine learning

Langley, P · 2000

Earlier work this paper cites.

The unreasonable effectiveness of data

Halevy, A., Norvig, P., and Pereira, F · 2009

Earlier work this paper cites.

Adam: A method for stochastic optimization

Original

Kingma, D. P. and Ba, J · 2014

Earlier work this paper cites.

Applying the roofline model

Ofenbeck, G., Steinmann, R., Caparros, V., Spampinato, D. G., and Püschel, M · 2014

Earlier work this paper cites.

Attention is all you need

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I · 2017

Earlier work this paper cites.

Pytorch: An imperative style, high-performance deep learning library, 2019

Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Köpf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S · 2019

Earlier work this paper cites.

Language models are unsupervised multitask learners

Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al · 2019

Earlier work this paper cites.

Language models are few-shot learners

Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al · 2020

Cited alongside, same era.

The pile: An 800gb dataset of diverse text for language modeling

Original

Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al · 2020

Cited alongside, same era.

Exploring the limits of transfer learning with a unified text-to-text transformer

Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J · 2020

Cited alongside, same era.

Extracting training data from large language models

Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T. B., Song, D., Erlingsson, U., et al · 2021

Cited alongside, same era.

Evaluating large language models trained on code

Original

Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al · 2021

Gptq: Accurate post-training quantization for generative pre-trained transformers

Original

Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D · 2022

Later among the works it cites.

The stack: 3 tb of permissively licensed source code

Original

Kocetkov, D., Li, R., Allal, L. B., Li, J., Mou, C., Ferrandis, C. M., Jernite, Y., Mitchell, M., Hughes, S., Wolf, T., et al · 2022

Later among the works it cites.

Fast inference from transformers via speculative decoding

Original

Leviathan, Y., Kalman, M., and Matias, Y · 2022

Later among the works it cites.

Nvidia RTX 4090 GPU architecture, 2022

NVIDIA · 2022

Later among the works it cites.

Accelerating large language model decoding with speculative sampling

Original

Chen, C., Borgeaud, S., Irving, G., Lespiau, J.-B., Sifre, L., and Jumper, J · 2023