Fetching the paper…
Reading the bibliography…
Large language models (LLMs) are increasingly integrated into many online services, yet they remain cost-prohibitive to deploy due to the requirement of expensive GPU instances.
Deep residual learning for image recognition, 2015
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2015
Earlier work this paper cites.
Branchynet: Fast inference via early exiting from deep neural networks
Surat Teerapittayanon, Bradley McDanel, and Hsiang-Tsung Kung · 2016
Earlier work this paper cites.
In-datacenter performance analysis of a tensor processing unit
Norman P Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, et al · 2017
Earlier work this paper cites.
A discourse-aware attention model for abstractive summarization of long documents
Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian · 2018
Earlier work this paper cites.
Tributary: spot-dancing for elastic services with latency { \{ SLOs } \}
Aaron Harlap, Andrew Chung, Alexey Tumanov, Gregory R Ganger, and Phillip B Gibbons · 2018
Earlier work this paper cites.
Balancing efficiency and fairness in heterogeneous gpu clusters for deep learning
Shubham Chaudhary, Ramachandran Ramjee, Muthian Sivathanu, Nipun Kwatra, and Srinidhi Viswanatha · 2020
Earlier work this paper cites.
{ \{ Heterogeneity-Aware } \} cluster scheduling policies for deep learning workloads
Deepak Narayanan, Keshav Santhanam, Fiodar Kazhamiaka, Amar Phanishayee, and Matei Zaharia · 2020
Earlier work this paper cites.
Big bird: Transformers for longer sequences
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al · 2020
Earlier work this paper cites.
Bert loses patience: Fast and robust inference with early exit
Wangchunshu Zhou, Canwen Xu, Tao Ge, Julian McAuley, Ke Xu, and Furu Wei · 2020
Earlier work this paper cites.
{ \{ INFaaS } \} : Automated model-less inference serving
Francisco Romero, Qian Li, Neeraja J Yadwadkar, and Christos Kozyrakis · 2021
Earlier work this paper cites.
Optimizing inference serving on serverless platforms
Ahsan Ali, Riccardo Pinciroli, Feng Yan, and Evgenia Smirni · 2022
Earlier work this paper cites.
Petals: Collaborative inference and fine-tuning of large models
Alexander Borzunov, Dmitry Baranchuk, Tim Dettmers, Max Ryabinin, Younes Belkada, Artem Chumachenko, Pavel Samygin, and Colin Raffel · 2022
Earlier work this paper cites.
Flashattention: Fast and memory-efficient exact attention with io-awareness
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré · 2022
Earlier work this paper cites.
Gptq: Accurate post-training quantization for generative pre-trained transformers
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh · 2022
Earlier work this paper cites.
Cocktail: A multidimensional optimization for model serving in cloud
Jashwant Raj Gunasekaran, Cyan Subhra Mishra, Prashanth Thinakaran, Bikash Sharma, Mahmut Taylan Kandemir, and Chita R Das · 2022
Earlier work this paper cites.
Srifty: Swift and thrifty distributed neural network training on the cloud
Liang Luo, Peter West, Pratyush Patel, Arvind Krishnamurthy, and Luis Ceze · 2022
Earlier work this paper cites.
Galvatron: Efficient transformer training over multiple gpus using automatic parallelism
Xupeng Miao, Yujie Wang, Youhe Jiang, Chunan Shi, Xiaonan Nie, Hailin Zhang, and Bin Cui · 2022
Earlier work this paper cites.
Zeroquant: Efficient and affordable post-training quantization for large-scale transformers
Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He · 2022
Earlier work this paper cites.
Orca: A distributed serving system for { \{ Transformer-Based } \} generative models
Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun · 2022
Earlier work this paper cites.
Sarathi: Efficient llm inference by piggybacking decodes with chunked prefills
Amey Agrawal, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S Gulavani, and Ramachandran Ramjee · 2023
Earlier work this paper cites.
Sparsegpt: Massive language models can be accurately pruned in one-shot, 2023
Elias Frantar and Dan Alistarh · 2023
Cited alongside, same era.
Hexgen: Generative inference of foundation model over heterogeneous decentralized environment
Youhe Jiang, Ran Yan, Xiaozhe Yao, Beidi Chen, and Binhang Yuan · 2023
Cited alongside, same era.
Efficient memory management for large language model serving with pagedattention
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica · 2023
Cited alongside, same era.
Fast inference from transformers via speculative decoding
Yaniv Leviathan, Matan Kalman, and Yossi Matias · 2023
Cited alongside, same era.
Awq: Activation-aware weight quantization for llm compression and acceleration
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han · 2023
Cited alongside, same era.
Inference without interference: Disaggregate llm inference for mixed downstream workloads
Cunchen Hu, Heyang Huang, Liangliang Xu, Xusheng Chen, Jiang Xu, Shuang Chen, Hao Feng, Chenxi Wang, Sa Wang, Yungang Bao, et al · 2024
Closest in time.
S3: Increasing gpu utilization during generative inference for higher throughput
Yunho Jin, Chun-Feng Wu, David Brooks, and Gu-Yeon Wei · 2024
Closest in time.
Speculative decoding with big little decoder
Sehoon Kim, Karttikeya Mangalam, Suhong Moon, Jitendra Malik, Michael W Mahoney, Amir Gholami, and Kurt Keutzer · 2024
Closest in time.
Andes: Defining and enhancing quality-of-experience in llm-based text streaming services
Jiachen Liu, Zhiyu Wu, Jae-Won Chung, Fan Lai, Myungjin Lee, and Mosharaf Chowdhury · 2024
Closest in time.
Reinventing search with a new ai-powered microsoft bing and edge, your copilot for the web, 2023
Yusuf Mehdi · 2024
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Spotserve: Serving generative large language models on preemptible instances
Xupeng Miao, Chunan Shi, Jiangfei Duan, Xiaoli Xi, Dahua Lin, Bin Cui, and Zhihao Jia · 2023
Cited alongside, same era.
Sdpipe: A semi-decentralized framework for heterogeneity-aware pipeline-parallel training
Xupeng Miao, Yining Shi, Zhi Yang, Bin Cui, and Zhihao Jia · 2023
Cited alongside, same era.
Gpt-4 technical report
OpenAI · 2023
Cited alongside, same era.
Splitwise: Efficient generative llm inference using phase splitting
Pratyush Patel, Esha Choukse, Chaojie Zhang, Íñigo Goiri, Aashaka Shah, Saeed Maleki, and Ricardo Bianchini · 2023
Cited alongside, same era.
Bamboo: Making preemptible instances resilient for affordable training of large { \{ DNNs } \}
John Thorpe, Pengzhan Zhao, Jonathan Eyolfson, Yifan Qiao, Zhihao Jia, Minjia Zhang, Ravi Netravali, and Guoqing Harry Xu · 2023
Cited alongside, same era.
Llama: Open and efficient foundation language models
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al · 2023
Cited alongside, same era.
Llama 2: Open foundation and fine-tuned chat models
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al · 2023
Cited alongside, same era.
Closest in time.
Copilot, 2023
Microsoft · 2024
Closest in time.
PuLP: A linear programming toolkit for python
Stuart Mitchell · 2024
Closest in time.
A10 gpu spec, 2024
Nvidia · 2024
Closest in time.
A100 gpu spec, 2024
Nvidia · 2024
Closest in time.
Gpus, 2024
Nvidia · 2024
Closest in time.
Chatgpt, 2022
OpenAI · 2024
Closest in time.
Supercharging search with generative ai, 2023
Elizabeth Reid · 2024
Closest in time.
Runpod, 2024
RunPod · 2024
Closest in time.
Accelerating self-attentions for llm serving with flashinfer, 2024
FlashInfer team · 2024
Closest in time.
Databricks: Training llms at scale with amd mi250 gpus
Abhi Venigalla · 2024
Closest in time.
H2o: Heavy-hitter oracle for efficient generative inference of large language models
Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al · 2024
Closest in time.
Response length perception and sequence scheduling: An llm-empowered llm inference pipeline
Zangwei Zheng, Xiaozhe Ren, Fuzhao Xue, Yang Luo, Xin Jiang, and Yang You · 2024
Closest in time.
Distserve: Disaggregating prefill and decoding for goodput-optimized large language model serving
Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang · 2024
Closest in time.
Towards optimal caching and model selection for large model inference
Banghua Zhu, Ying Sheng, Lianmin Zheng, Clark Barrett, Michael Jordan, and Jiantao Jiao · 2024
Closest in time.