Fetching the paper…
Reading the bibliography…
Transformer-based large language models (LLMs) exhibit impressive performance in generative tasks but also introduce significant challenges in real-world serving due to inefficient use of the expensive, computation-optimized accelerators.
Roofline: An insightful visual performance model for multicore architectures
Samuel Williams, Andrew Waterman, and David Patterson · 2009
Earlier work this paper cites.
Scaling with design constraints: Predicting the future of big chips
Wei Huang, Karthick Rajamani, Mircea R Stan, and Kevin Skadron · 2011
Earlier work this paper cites.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin · 2017
Earlier work this paper cites.
Low latency rnn inference with cellular batching
Pin Gao, Lingfan Yu, Yongwei Wu, and Jinyang Li · 2018
Earlier work this paper cites.
Generating long sequences with sparse transformers
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever · 2019
Earlier work this paper cites.
Blockwise self-attention for long document understanding
Jiezhong Qiu, Hao Ma, Omer Levy, Scott Wen-tau Yih, Sinong Wang, and Jie Tang · 2019
Earlier work this paper cites.
Bp-transformer: Modelling long-range context via binary partitioning
Zihao Ye, Qipeng Guo, Quan Gan, Xipeng Qiu, and Zheng Zhang · 2019
Earlier work this paper cites.
https://network.nvidia.com/files/doc-2020/ethernet-adapters-programming-manual.pdf
Mellanox adapters programmer’s reference manual · 2020
Earlier work this paper cites.
Longformer: The long-document transformer
Iz Beltagy, Matthew E Peters, and Arman Cohan · 2020
Earlier work this paper cites.
Newton: A dram-maker’s accelerator-in-memory (aim) architecture for machine learning
Mingxuan He, Choungki Song, Ilkon Kim, Chunseok Jeong, Seho Kim, Il Park, Mithuna Thottethodi, and T. N. Vijaykumar · 2020
Earlier work this paper cites.
Reformer: The efficient transformer
Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya · 2020
Earlier work this paper cites.
Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro · 2020
Earlier work this paper cites.
Retransformer: Reram-based processing-in-memory architecture for transformer acceleration
Xiaoxuan Yang, Bonan Yan, Hai Li, and Yiran Chen · 2020
Earlier work this paper cites.
Lazy batching: An sla-aware batching system for cloud machine learning inference
Y. Choi, Y. Kim, and M. Rhu · 2021
Earlier work this paper cites.
Turbotransformers: An efficient gpu serving system for transformer models
Jiarui Fang, Yang Yu, Chengduo Zhao, and Jie Zhou · 2021
Earlier work this paper cites.
Lora: Low-rank adaptation of large language models, 2021
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen · 2021
Earlier work this paper cites.
In-memory computing based accelerator for transformer networks for long sequences
Ann Franchesca Laguna, Arman Kazemi, Michael Niemier, and X. Sharon Hu · 2021
Earlier work this paper cites.
Efficient content-based sparse attention with routing transformers
Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier · 2021
Earlier work this paper cites.
Deepspeed inference: Enabling efficient inference of transformer models at unprecedented scale, 2022
Reza Yazdani Aminabadi, Samyam Rajbhandari, Minjia Zhang, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Jeff Rasley, Shaden Smith, Olatunji Ruwase, and Yuxiong He · 2022
Cited alongside, same era.
Aquabolt-xl hbm2-pim, lpddr5-pim with in-memory processing, and axdimm with acceleration buffer
Jin Hyun Kim, Shin-Haeng Kang, Sukhan Lee, Hyeonsu Kim, Yuhwan Ro, Seungwon Lee, David Wang, Jihyun Choi, Jinin So, YeonGon Cho, JoonHo Song, Jeonghyeon Cho, Kyomin Sohn, and Nam Sung Kim · 2022
Cited alongside, same era.
System architecture and software stack for gddr6-aim
Yongkee Kwon, Kornijcuk Vladimir, Nahsung Kim, Woojae Shin, Jongsoon Won, Minkyu Lee, Hyunha Joo, Haerang Choi, Guhyun Kim, Byeongju An, Jeongbin Kim, Jaewook Lee, Ilkon Kim, Jaehan Park, Chanwook Park, Yosub Song, Byeongsu Yang, Hyungdeok Lee, Seho Kim, Daehan Kwon, Seongju Lee, Kyuyoung Kim, Sanghoon Oh, Joonhong Park, Gimoon Hong, Dongyoon Ka, Kyudong Hwang, Jeongje Park, Kyeongpil Kang, Jungyeon Kim, Junyeol Jeon, Myeongjun Lee, Minyoung Shin, Minhwan Shin, Jaekyung Cha, Changson Jung, Kijoon Chang, Chunseok Jeong, Euicheol Lim, Il Park, Junhyun Chun, and Sk Hynix · 2022
Cited alongside, same era.
E-batch: Energy-efficient and high-throughput rnn batching
Franyell Silfa, Jose Maria Arnau, and Antonio González · 2022
Cited alongside, same era.
Ring attention with blockwise transformers for near-infinite context, 2023
Hao Liu, Matei Zaharia, and Pieter Abbeel · 2023
Later among the works it cites.
Online speculative decoding, 2023
Xiaoxuan Liu, Lanxiang Hu, Peter Bailis, Ion Stoica, Zhijie Deng, Alvin Cheung, and Hao Zhang · 2023
Later among the works it cites.
Deja vu: Contextual sparsity for efficient llms at inference time
Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Re, et al · 2023
Later among the works it cites.
Specinfer: Accelerating generative large language model serving with speculative inference and token tree verification, 2023
Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, Chunan Shi, Zhuoming Chen, Daiyaan Arfeen, Reyna Abhyankar, and Zhihao Jia · 2023
Later among the works it cites.
Flexgen: High-throughput generative inference of large language models with a single gpu
Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang · 2023
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Orca: A distributed serving system for Transformer-Based generative models
Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun · 2022
Cited alongside, same era.
Opt: Open pre-trained transformer language models, 2022
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer · 2022
Cited alongside, same era.
https://github.com/Azure/AzurePublicDataset/blob/master/AzureLLMInferenceDataset2023.md
Azure llm inference trace 2023 · 2023
Cited alongside, same era.
Sarathi: Efficient llm inference by piggybacking decodes with chunked prefills, 2023
Amey Agrawal, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, and Ramachandran Ramjee · 2023
Cited alongside, same era.
Gqa: Training generalized multi-query transformer models from multi-head checkpoints, 2023
Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai · 2023
Cited alongside, same era.
A survey on processing-in-memory techniques: Advances and challenges
Kazi Asifuzzaman, Narasinga Rao Miniskar, Aaron R Young, Frank Liu, and Jeffrey S Vetter · 2023
Cited alongside, same era.
Flash-decoding for long-context inference
Tri Dao, Daniel Haziza, Francisco Massa, and Grigory Sizov · 2023
Cited alongside, same era.
Fast inference of mixture-of-experts language models with offloading, 2023
Artyom Eliseev and Denis Mazur · 2023
Cited alongside, same era.
Later among the works it cites.
X-former: In-memory acceleration of transformers
Shrihari Sridharan, Jacob R. Stevens, Kaushik Roy, and Anand Raghunathan · 2023
Later among the works it cites.
Llama: Open and efficient foundation language models, 2023
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample · 2023
Later among the works it cites.
Fast distributed inference serving for large language models, 2023
Bingyang Wu, Yinmin Zhong, Zili Zhang, Gang Huang, Xuanzhe Liu, and Xin Jin · 2023
Later among the works it cites.
Llama 3 model card
AI@Meta · 2024
Closest in time.
Hexgen-2: Disaggregated generative inference of LLMs in heterogeneous environment
Anonymous · 2024
Closest in time.
Ai and memory wall
Amir Gholami, Zhewei Yao, Sehoon Kim, Coleman Hooper, Michael W Mahoney, and Kurt Keutzer · 2024
Closest in time.
Splitwise: Efficient generative llm inference using phase splitting, 2024
Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini · 2024
Closest in time.
Mooncake: A kvcache-centric disaggregated architecture for llm serving
Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu · 2024
Closest in time.
Efficient streaming language models with attention sinks, 2024
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis · 2024
Closest in time.
Moe-infinity: Offloading-efficient moe model serving, 2024
Leyang Xue, Yao Fu, Zhan Lu, Luo Mai, and Mahesh Marina · 2024
Closest in time.
Challenges and opportunities to enable large-scale computing via heterogeneous chiplets
Zhuoping Yang, Shixin Ji, Xingzhen Chen, Jinming Zhuang, Weifeng Zhang, Dharmesh Jani, and Peipei Zhou · 2024
Closest in time.
DistServe: Disaggregating prefill and decoding for goodput-optimized large language model serving
Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang · 2024
Closest in time.