EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test

Distilling the knowledge in a neural network

Original

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015 · 2015

Earlier work this paper cites.

Abstractive text summarization using sequence-to-sequence rnns and beyond

Original

Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. 2016 · 2016

Earlier work this paper cites.

Quantized neural networks: Training neural networks with low precision weights and activations

Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. 2018 · 2018

Earlier work this paper cites.

Blockwise parallel decoding for deep autoregressive models

Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. 2018 · 2018

Earlier work this paper cites.

Q8bert: Quantized 8bit bert

Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. 2019 · 2019

Earlier work this paper cites.

Q-bert: Hessian based ultra low precision quantization of bert

Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. 2020 · 2020

Earlier work this paper cites.

Gobo: Quantizing attention-based nlp models for low latency and energy efficient inference

Ali Hadi Zadeh, Isak Edo, Omar Mohamed Awad, and Andreas Moshovos. 2020 · 2020

Earlier work this paper cites.

Evaluating large language models trained on code

Original

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021 · 2021

Earlier work this paper cites.

Training verifiers to solve math word problems

Original

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021 · 2021

Earlier work this paper cites.

I-bert: Integer-only bert quantization

Sehoon Kim, Amir Gholami, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. 2021 · 2021

Earlier work this paper cites.

Instantaneous grammatical error correction with shallow aggressive decoding

Original

Xin Sun, Tao Ge, Furu Wei, and Houfeng Wang. 2021 · 2021

Earlier work this paper cites.

Accelerating large language model decoding with speculative sampling

Original

Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. 2023 · 2023

Earlier work this paper cites.

Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023 · 2023

Earlier work this paper cites.

Enhancing chat language models by scaling high-quality instructional conversations

Original

Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. 2023 · 2023

Cited alongside, same era.

Assisted generation: a new direction toward low-latency text generation

Joao Gante. 2023 · 2023

Cited alongside, same era.

Speed: Speculative pipelined execution for efficient decoding

Original

Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Hasan Genc, Kurt Keutzer, Amir Gholami, and Sophia Shao. 2023 · 2023

Cited alongside, same era.

Efficient memory management for large language model serving with pagedattention

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023 · 2023

Cited alongside, same era.

Fast inference from transformers via speculative decoding

Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023 · 2023

Glide with a cape: A low-hassle method to accelerate speculative decoding

Original

Cunxiao Du, Jing Jiang, Xu Yuanchen, Jiawei Wu, Sicheng Yu, Yongqi Li, Shenggui Li, Kai Xu, Liqiang Nie, Zhaopeng Tu, et al. 2024 · 2024

Later among the works it cites.

The Llama 3 herd of models

Original

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024 · 2024

Later among the works it cites.

Layer skip: Enabling early exit inference and self-speculative decoding

Original

Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, et al. 2024 · 2024

Later among the works it cites.

Break the sequential dependency of LLM inference using lookahead decoding

Original

Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. 2024 · 2024

Later among the works it cites.