AQUA: Network-Accelerated Memory Offloading for LLMs in Scale-Up GPU Domains

Salus: Fine-Grained GPU Sharing Primitives for Deep Learning Applications

Original

Peifeng Yu and Mosharaf Chowdhury. 2019 · 1902

Earlier work this paper cites.

Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

Original

Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2020 · 1909

Earlier work this paper cites.

A Neural Probabilistic Language Model. In Advances in Neural Information Processing Systems , T. Leen, T. Dietterich, and V. Tresp (Eds.), Vol. 13. MIT Press

Yoshua Bengio, Réjean Ducharme, and Pascal Vincent. 2000 · 2000

Earlier work this paper cites.

Efficient Estimation of Word Representations in Vector Space

Original

Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013 · 2013

Earlier work this paper cites.

Reducing Internet Latency: A Survey of Techniques and Their Merits

Bob Briscoe, Anna Brunstrom, Andreas Petlund, David Hayes, David Ros, Ing-Jyh Tsang, Stein Gjessing, Gorry Fairhurst, Carsten Griwodz, and Michael Welzl. 2016 · 2014

Earlier work this paper cites.

Jupiter Rising: A Decade of Clos Topologies and Centralized Control in Google’s Datacenter Network. In Sigcomm ’15

Arjun Singh, Joon Ong, Amit Agarwal, Glen Anderson, Ashby Armistead, Roy Bannon, Seb Boving, Gaurav Desai, Bob Felderman, Paulie Germano, Anand Kanagala, Jeff Provost, Jason Simmons, Eiichi Tanda, Jim Wanderer, Urs Hölzle, Stephen Stuart, and Amin Vahdat. 2015 · 2015

Earlier work this paper cites.

Efficient Memory Disaggregation with Infiniswap. In 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 17) . USENIX Association, Boston, MA, 649–667

Juncheng Gu, Youngmoon Lee, Yiwen Zhang, Mosharaf Chowdhury, and Kang G. Shin. 2017 · 2017

Earlier work this paper cites.

Attention is All you Need. In Advances in Neural Information Processing Systems , I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.), Vol. 30. Curran Associates, Inc

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017 · 2017

Earlier work this paper cites.

Gandiva: Introspective Cluster Scheduling for Deep Learning. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18) . USENIX Association, Carlsbad, CA, 595–610

Wencong Xiao, Romil Bhardwaj, Ramachandran Ramjee, Muthian Sivathanu, Nipun Kwatra, Zhenhua Han, Pratyush Patel, Xuan Peng, Hanyu Zhao, Quanlu Zhang, Fan Yang, and Lidong Zhou. 2018 · 2018

Earlier work this paper cites.

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Original

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 · 2019

Earlier work this paper cites.

Can far memory improve job throughput?. In Proceedings of the Fifteenth European Conference on Computer Systems (Heraklion, Greece) (EuroSys ’20) . Association for Computing Machinery, New York, NY, USA, Article 14, 16 pages

Emmanuel Amaro, Christopher Branner-Augmon, Zhihong Luo, Amy Ousterhout, Marcos K. Aguilera, Aurojit Panda, Sylvia Ratnasamy, and Scott Shenker. 2020 · 2020

Earlier work this paper cites.

PipeSwitch: Fast Pipelined Context Switching for Deep Learning Applications. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20) . USENIX Association, 499–514

Zhihao Bai, Zhen Zhang, Yibo Zhu, and Xin Jin. 2020 · 2020

Earlier work this paper cites.

Serving DNNs like Clockwork: Performance Predictability from the Bottom Up. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20) . USENIX Association, 443–462

Arpan Gujarati, Reza Karimi, Safya Alzayat, Wei Hao, Antoine Kaufmann, Ymir Vigfusson, and Jonathan Mace. 2020 · 2020

Earlier work this paper cites.

The future of AI is Wafer-Scale

Cerebras 2021 · 2021

Earlier work this paper cites.

Intel Gaudi AI accelerator

Intel Gaudi AI accelerator 2021 · 2021

Earlier work this paper cites.

Nvidia DGX Systems

Nvidia DGX Systems 2021 · 2021

Earlier work this paper cites.

DeepSpeed-inference: enabling efficient inference of transformer models at unprecedented scale. In Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis (Dallas, Texas) (SC ’22) . IEEE Press, Article 46, 15 pages

Reza Yazdani Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Minjia Zhang, Jeff Rasley, and Yuxiong He. 2022 · 2022

Earlier work this paper cites.

Memory Harvesting in Multi-GPU Systems with Hierarchical Unified Virtual Memory. In 2022 USENIX Annual Technical Conference (USENIX ATC 22) . USENIX Association, Carlsbad, CA, 625–638

Sangjin Choi, Taeksoo Kim, Jinwoo Jeong, Rachata Ausavarungnirun, Myeongjae Jeon, Youngjin Kwon, and Jeongseob Ahn. 2022 · 2022

Earlier work this paper cites.

High-Resolution Image Synthesis with Latent Diffusion Models

Original

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022 · 2022

Earlier work this paper cites.

Orca: A Distributed Serving System for Transformer-Based Generative Models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22) . USENIX Association, Carlsbad, CA, 521–538

Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022 · 2022

Earlier work this paper cites.

Host Congestion Control. In Proceedings of the ACM SIGCOMM 2023 Conference (<conf-loc>, <city>New York</city>, <state>NY</state>, <country>USA</country>, </conf-loc>) (ACM SIGCOMM ’23) . Association for Computing Machinery, New York, NY, USA, 275–287

Saksham Agarwal, Arvind Krishnamurthy, and Rachit Agarwal. 2023 · 2023

Cited alongside, same era.

TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings

Original

Norman P. Jouppi, George Kurian, Sheng Li, Peter Ma, Rahul Nagarajan, Lifeng Nai, Nishant Patil, Suvinay Subramanian, Andy Swing, Brian Towles, Cliff Young, Xiang Zhou, Zongwei Zhou, and David Patterson. 2023 · 2023

Cited alongside, same era.

AudioGen: Textually Guided Audio Generation

Original

Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre Défossez, Jade Copet, Devi Parikh, Yaniv Taigman, and Yossi Adi. 2023 · 2023

Cited alongside, same era.

NVLink & NVSwitch: Fastest HPC Data Center Platform

NVIDIA. 2024 · 2024

Closest in time.

NVIDIA B200

NVIDIA Corporation. 2024a · 2024

Closest in time.

NVIDIA DGX A100 Datasheet

NVIDIA Corporation. 2024b · 2024

Closest in time.

NVIDIA H100 Tensor Core GPU Datasheet

NVIDIA Corporation. 2024c · 2024

Closest in time.

OpenAI API

OpenAI. 2024a · 2024

Closest in time.

OpenAI batch API

OpenAI. 2024b · 2024

Closest in time.

OpenAI Customer Stories

OpenAI. 2024c · 2024

Closest in time.