Fetching the paper…
Reading the bibliography…
The high computational and memory requirements of generative large language models (LLMs) make it challenging to serve them cheaply.
Tensorflow: A system for large-scale machine learning
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng · 2016
Earlier work this paper cites.
https://developer.nvidia.com/cublas , 2016
Dense Linear Algebra on GPUs · 2016
Earlier work this paper cites.
Clipper: A Low-Latency online prediction serving system
Daniel Crankshaw, Xin Wang, Guilio Zhou, Michael J. Franklin, Joseph E. Gonzalez, and Ion Stoica · 2017
Earlier work this paper cites.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin · 2017
Earlier work this paper cites.
Gpipe: Efficient training of giant neural networks using pipeline parallelism
Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Xu Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen · 2019
Earlier work this paper cites.
Beyond data and model parallelism for deep neural networks
Zhihao Jia, Matei Zaharia, and Alex Aiken · 2019
Earlier work this paper cites.
Parity models: erasure-coded resilience for prediction serving systems
Jack Kosaian, KV Rashmi, and Shivaram Venkataraman · 2019
Earlier work this paper cites.
Pipedream: Generalized pipeline parallelism for dnn training
Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R. Devanur, Gregory R. Ganger, Phillip B. Gibbons, and Matei Zaharia · 2019
Earlier work this paper cites.
fairseq: A fast, extensible toolkit for sequence modeling
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli · 2019
Earlier work this paper cites.
Language models are unsupervised multitask learners
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al · 2019
Earlier work this paper cites.
Nexus: A gpu cluster engine for accelerating dnn-based video analysis
Haichen Shen, Lequn Chen, Yuchen Jin, Liangyu Zhao, Bingyu Kong, Matthai Philipose, Arvind Krishnamurthy, and Ravi Sundaram · 2019
Earlier work this paper cites.
Megatron-lm: Training multi-billion parameter language models using model parallelism
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro · 2019
Earlier work this paper cites.
Mark: Exploiting cloud services for cost-effective, slo-aware machine learning inference serving
Chengliang Zhang, Minchen Yu, Wei Wang, and Feng Yan · 2019
Earlier work this paper cites.
Pretraining-based natural language generation for text summarization
Haoyu Zhang, Jianjun Xu, and Ji Wang · 2019
Earlier work this paper cites.
Batch: Machine learning inference serving on serverless platforms with adaptive batching
Ahsan Ali, Riccardo Pinciroli, Feng Yan, and Evgenia Smirni · 2020
Earlier work this paper cites.
Language models are few-shot learners, 2020
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei · 2020
Earlier work this paper cites.
Serving DNNs like clockwork: Performance predictability from the bottom up
Arpan Gujarati, Reza Karimi, Safya Alzayat, Wei Hao, Antoine Kaufmann, Ymir Vigfusson, and Jonathan Mace · 2020
Earlier work this paper cites.
Gshard: Scaling giant models with conditional computation and automatic sharding
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen · 2020
Cited alongside, same era.
Serverless in the wild: Characterizing and optimizing the serverless workload at a large cloud provider
Mohammad Shahrad, Rodrigo Fonseca, Inigo Goiri, Gohar Chaudhry, Paul Batum, Jason Cooke, Eduardo Laureano, Colby Tresness, Mark Russinovich, and Ricardo Bianchini · 2020
Cited alongside, same era.
Optimizing prediction serving on low-latency serverless dataflow
Vikram Sreekanti, Harikaran Subbaraj, Chenggang Wu, Joseph E. Gonzalez, and Joseph M. Hellerstein · 2020
Cited alongside, same era.
https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__DEVICE.html , 2021
Cuda ipc · 2021
Cited alongside, same era.
https://github.com/NVIDIA/FasterTransformer , 2021
Nvidia fastertransformer · 2021
Cocktail: A multidimensional optimization for model serving in cloud
Jashwant Raj Gunasekaran, Cyan Subhra Mishra, Prashanth Thinakaran, Bikash Sharma, Mahmut Taylan Kandemir, and Chita R. Das · 2022
Later among the works it cites.
Tetris: Memory-efficient serverless inference through tensor sharing
Jie Li, Laiping Zhao, Yanan Yang, Kunlin Zhan, and Keqiu Li · 2022
Later among the works it cites.
Bamboo: Making preemptible instances resilient for affordable training of large dnns
John Thorpe, Pengzhan Zhao, Jonathan Eyolfson, Yifan Qiao, Zhihao Jia, Minjia Zhang, Ravi Netravali, and Guoqing Harry Xu · 2022
Later among the works it cites.
Unity: Accelerating DNN training through joint optimization of algebraic transformations and parallelization
Colin Unger, Zhihao Jia, Wei Wu, Sina Lin, Mandeep Baines, Carlos Efrain Quintero Narvaez, Vinay Ramakrishnaiah, Nirmal Prajapati, Patrick S. McCormick, Jamaludin Mohd-Yusof, Xi Luo, Dheevatsa Mudigere, Jongsoo Park, Misha Smelyanskiy, and Alex Aiken · 2022
Later among the works it cites.
Orca: A distributed serving system for Transformer-Based generative models
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Cited alongside, same era.
https://developer.nvidia.com/nccl , 2021
Nvidia nccl · 2021
Cited alongside, same era.
Turbotransformers: an efficient GPU serving system for transformer models
Jiarui Fang, Yang Yu, Chengduo Zhao, and Jie Zhou · 2021
Cited alongside, same era.
What makes good in-context examples for gpt- 3 3 ?
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen · 2021
Cited alongside, same era.
Memory-efficient pipeline-parallel DNN training
Deepak Narayanan, Amar Phanishayee, Kaiyu Shi, Xie Chen, and Matei Zaharia · 2021
Cited alongside, same era.
INFaaS: Automated model-less inference serving
Francisco Romero, Qian Li, Neeraja J. Yadwadkar, and Christos Kozyrakis · 2021
Cited alongside, same era.
Piper: Multidimensional planner for dnn parallelization
Jakub M Tarnawski, Deepak Narayanan, and Amar Phanishayee · 2021
Cited alongside, same era.
Lightseq: A high performance inference library for transformers
Xiaohui Wang, Ying Xiong, Yang Wei, Mingxuan Wang, and Lei Li · 2021
Cited alongside, same era.
Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun · 2022
Later among the works it cites.
Opt: Open pre-trained transformer language models
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al · 2022
Later among the works it cites.
Alpa: Automating inter- and intra-operator parallelism for distributed deep learning
Lianmin Zheng, Zhuohan Li, Hao Zhang, Yonghao Zhuang, Zhifeng Chen, Yanping Huang, Yida Wang, Yuanzhong Xu, Danyang Zhuo, Eric P. Xing, Joseph E. Gonzalez, and Ion Stoica · 2022
Later among the works it cites.
https://vllm.ai , 2023
vllm: Easy, fast, and cheap llm serving with pagedattention · 2023
Closest in time.
Deft: Slo-driven preemptive scheduling for containerized dnn serving
Yitian Hao, Wenqing Wu, Ziyi Zhang, Yuyang Huang, Chen Wang, Jun Duan, and Junchen Jiang · 2023
Closest in time.
Alpaserve: Statistical multiplexing with model parallelism for deep learning serving
Zhuohan Li, Lianmin Zheng, Yinmin Zhong, Vincent Liu, Ying Sheng, Xin Jin, Yanping Huang, Zhifeng Chen, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica · 2023
Closest in time.
Galvatron: Efficient transformer training over multiple gpus using automatic parallelism
Xupeng Miao, Yujie Wang, Youhe Jiang, Chunan Shi, Xiaonan Nie, Hailin Zhang, and Bin Cui · 2023
Closest in time.
Gpt-4 technical report, 2023
OpenAI · 2023
Closest in time.
High-throughput generative inference of large language models with a single GPU
Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Daniel Y. Fu, Zhiqiang Xie, Beidi Chen, Clark W. Barrett, Joseph E. Gonzalez, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang · 2023
Closest in time.
Llama: Open and efficient foundation language models
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al · 2023
Closest in time.
Snape: Reliable and low-cost computing with mixture of spot and on-demand vms
Fangkai Yang, Lu Wang, Zhenyu Xu, Jue Zhang, Liqun Li, Bo Qiao, Camille Couturier, Chetan Bansal, Soumya Ram, Si Qin, et al · 2023
Closest in time.
SkyPilot: An intercloud broker for sky computing
Zongheng Yang, Zhanghao Wu, Michael Luo, Wei-Lin Chiang, Romil Bhardwaj, Woosuk Kwon, Siyuan Zhuang, Frank Sifei Luan, Gautam Mittal, Scott Shenker, and Ion Stoica · 2023
Closest in time.
SHEPHERD: Serving DNNs in the wild
Hong Zhang, Yupeng Tang, Anurag Khandelwal, and Ion Stoica · 2023
Closest in time.