Towards Fast Multilingual LLM Inference: Speculative Decoding and Specialized Drafters

Fast transformer decoding: One write-head is all you need

Original

Noam Shazeer. 2019 · 1911

Earlier work this paper cites.

Speculative computation, parallelism, and functional programming

F Warren Burton. 1985 · 1985

Earlier work this paper cites.

Latency lags bandwith

David A Patterson. 2004 · 2004

Earlier work this paper cites.

Findings of the 2014 workshop on statistical machine translation

Ondřej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, et al. 2014 · 2014

Earlier work this paper cites.

Findings of the 2016 conference on machine translation (wmt16)

Ondrej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, et al. 2016 · 2016

Earlier work this paper cites.

Gaussian error linear units (gelus)

Original

Dan Hendrycks and Kevin Gimpel. 2016 · 2016

Earlier work this paper cites.

Sequence-level knowledge distillation

Original

Yoon Kim and Alexander M Rush. 2016 · 2016

Earlier work this paper cites.

Decoupled weight decay regularization

Original

Ilya Loshchilov and Frank Hutter. 2017 · 2017

Earlier work this paper cites.

Blockwise parallel decoding for deep autoregressive models

Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. 2018 · 2018

Earlier work this paper cites.

Exploring the limits of transfer learning with a unified text-to-text transformer

Original

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019 · 2019

Earlier work this paper cites.

Train big, then compress: Rethinking model size for efficient training and inference of transformers

Zhuohan Li, Eric Wallace, Sheng Shen, Kevin Lin, Kurt Keutzer, Dan Klein, and Joey Gonzalez. 2020 · 2020

Earlier work this paper cites.

Training verifiers to solve math word problems

Original

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021 · 2021

Earlier work this paper cites.

Discovering representation sprachbund for multilingual pre-training

Original

Yimin Fan, Yaobo Liang, Alexandre Muzio, Hany Hassan, Houqiang Li, Ming Zhou, and Nan Duan. 2021 · 2021

Earlier work this paper cites.

Jparacrawl v3. 0: A large-scale english-japanese parallel corpus

Original

Makoto Morishita, Katsuki Chousa, Jun Suzuki, and Masaaki Nagata. 2022 · 2022

Earlier work this paper cites.

Confident adaptive language modeling

Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Tran, Yi Tay, and Donald Metzler. 2022 · 2022

Earlier work this paper cites.

Gpt-4 technical report

Original

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023 · 2023

Earlier work this paper cites.

Do all languages cost the same? tokenization in the era of commercial language models

Original

Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David R Mortensen, Noah A Smith, and Yulia Tsvetkov. 2023 · 2023

Cited alongside, same era.

Fast and robust early-exiting framework for autoregressive language models with synchronized parallel decoding

Original

Sangmin Bae, Jongwoo Ko, Hwanjun Song, and Se-Young Yun. 2023 · 2023

Cited alongside, same era.

Accelerating large language model decoding with speculative sampling

Original

Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. 2023 · 2023

Cited alongside, same era.

Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023 · 2023

Cited alongside, same era.

Medusa: Simple llm inference acceleration framework with multiple decoding heads

Original

Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao. 2024 · 2024

Closest in time.

Layer skip: Enabling early exit inference and self-speculative decoding

Original

Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, et al. 2024 · 2024

Closest in time.

Break the sequential dependency of llm inference using lookahead decoding

Original

Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. 2024 · 2024

Closest in time.

Better & faster large language models via multi-token prediction

Original

Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, and Gabriel Synnaeve. 2024 · 2024

Closest in time.