2024

Trillion Parameter AI Serving Infrastructure for Scientific Discovery: A Survey and Vision

Hudson, Nathaniel, Pauloski, J. Gregory, Baughman, Matt et al.

Understand

Deep learning methods are transforming research, enabling new techniques, and ultimately leading to new discoveries.

As the demand for more capable AI models continues to grow, we are now entering an era of Trillion Parameter Models (TPM), or models with more than a trillion parameters -- such as Huawei's PanGu-$\Sigma$.
We describe a vision for the ecosystem of TPM users and providers that caters to the specific needs of the scientific community.
We then outline the significant technical challenges and open problems in system design for serving TPMs to enable scientific research and discovery.

Built on

PyTorch: An Imperative Style, High-Performance Deep Learning Library
Original
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, et al · 1912
Earlier work this paper cites.
ρ \rho - σ \sigma - π \pi Analysis. A Method for the Correlation of Biological Activity and Chemical Structure
Corwin. Hansch and Toshio. Fujita. 1964 · 1964
Earlier work this paper cites.
Long Short-term Memory
Sepp Hochreiter and Jürgen Schmidhuber. 1997 · 1997
Earlier work this paper cites.
Scaling Laws for Neural Language Models
Original
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020 · 2001
Earlier work this paper cites.
Active Learning with Support Vector Machines in the Drug Discovery Process
Manfred K. Warmuth, Jun Liao, Gunnar Rätsch, Michael Mathieson, Santosh Putta, and Christian Lemmen. 2003 · 2003
Earlier work this paper cites.
Optimizing feedforward artificial neural network architecture
PG Benardos and G-C Vosniakos. 2007 · 2007
Earlier work this paper cites.
Quantitative Structure–Property Relationship Modeling of Diverse Materials Properties
Tu Le, V. Chandana Epa, Frank R. Burden, and David A. Winkler. 2012 · 2012
Earlier work this paper cites.
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) . ACL, 1724–1734
Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014 · 2014
Earlier work this paper cites.
Caffe: Convolutional Architecture for Fast Feature Embedding
Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. 2014 · 2014
Earlier work this paper cites.
Genetic algorithm for neural network architecture optimization. In 2016 3rd International conference on logistics operations management (GOL) . IEEE Computer Society, 1–4
Mohammed Amine Janati Idrissi, Hassan Ramchoun, Youssef Ghanou, and Mohamed Ettaouil. 2016 · 2016
Earlier work this paper cites.
Mastering the game of Go with deep neural networks and tree search
David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, et al · 2016
Earlier work this paper cites.
Clipper: A Low-Latency Online Prediction Serving System. In 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 17) . USENIX Association, 613–627
Daniel Crankshaw, Xin Wang, Guilio Zhou, Michael J. Franklin, Joseph E. Gonzalez, and Ion Stoica. 2017 · 2017
Earlier work this paper cites.
The Exascale Computing Project
Paul Messina. 2017 · 2017
Earlier work this paper cites.
Towards unified data and lifecycle management for deep learning. In 2017 IEEE 33rd International Conference on Data Engineering (ICDE) . IEEE Computer Society, 571–582
Hui Miao, Ang Li, Larry S Davis, and Amol Deshpande. 2017 · 2017
Earlier work this paper cites.
graph2vec: Learning distributed representations of graphs
Original
Annamalai Narayanan, Mahinthan Chandramohan, Rajasekar Venkatesan, Lihui Chen, Yang Liu, and Shantanu Jaiswal. 2017 · 2017
Earlier work this paper cites.
Attention is All you Need. In Advances in Neural Information Processing Systems , I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.), Vol. 30. Curran Associates, Inc
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017 · 2017
Earlier work this paper cites.
Automatic Chemical Design Using a Data-Driven Continuous Representation of Molecules
Rafael Gómez-Bombarelli, Jennifer N. Wei, David Duvenaud, José Miguel Hernández-Lobato, Benjamín Sánchez-Lengeling, et al · 2018
Earlier work this paper cites.
Progressive neural architecture search. In Proceedings of the European conference on computer vision (ECCV) . Springer, Cham, 19–34
Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, et al · 2018
Earlier work this paper cites.
Neural architecture optimization
Renqian Luo, Fei Tian, Tao Qin, Enhong Chen, and Tie-Yan Liu. 2018 · 2018
Earlier work this paper cites.
Ray: A distributed framework for emerging AI applications. In 13th USENIX symposium on operating systems design and implementation (OSDI 18) . USENIX Association, 561–577
Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, et al · 2018
Earlier work this paper cites.
Accelerating the machine learning lifecycle with MLflow
Matei Zaharia, Andrew Chen, Aaron Davidson, Ali Ghodsi, Sue Ann Hong, Andy Konwinski, Siddharth Murching, et al · 2018
Earlier work this paper cites.

Similar

DLHub: Model and data serving for science. In 2019 IEEE International Parallel and Distributed Processing Symposium (IPDPS) . IEEE, 283–292
Ryan Chard, Zhuozhao Li, Kyle Chard, Logan Ward, Yadu Babuji, Anna Woodard, Steven Tuecke, et al · 2019
Cited alongside, same era.
Transfer Learning to Model Inertial Confinement Fusion Experiments
K. D. Humbird, J. L. Peterson, B. K. Spears, and R. G. McClarren. 2020 · 2019
Cited alongside, same era.
Automated machine learning: methods, systems, challenges
Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren. 2019 · 2019
Cited alongside, same era.
On neural architecture search for resource-constrained hardware platforms
Qing Lu, Weiwen Jiang, Xiaowei Xu, Yiyu Shi, and Jingtong Hu. 2019 · 2019
Cited alongside, same era.
funcX: A Federated Function Serving Fabric for Science. In Proceedings of the 29th International Symposium on High-Performance Parallel and Distributed Computing . ACM
Ryan Chard, Yadu Babuji, Zhuozhao Li, Tyler Skluzacek, Anna Woodard, Ben Blaiszik, Ian Foster, and Kyle Chard. 2020 · 2020
Cited alongside, same era.
word2vec, node2vec, graph2vec, x2vec: Towards a theory of vector embeddings of structured data. In Proceedings of the 39th ACM SIGMOD-SIGACT-SIGAI Symposium on Principles of Database Systems . 1–16
Martin Grohe. 2020 · 2020
Cited alongside, same era.
A machine learning Automated Recommendation Tool for synthetic biology
Tijana Radivojević, Zak Costello, Kenneth Workman, and Hector Garcia Martin. 2020 · 2020
Cited alongside, same era.
ZeRO: Memory Optimizations toward Training Trillion Parameter Models. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC ’20) . IEEE Press, Article 20, 16 pages
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020 · 2020
Cited alongside, same era.
Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations . ACL, Online, 38–45
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, et al · 2020
Cited alongside, same era.
Improving the accuracy-latency trade-off of edge-cloud computation offloading for deep learning services. In 2020 IEEE Globecom Workshops . IEEE, 1–6
Xiaobo Zhao, Minoo Hosseinzadeh, Nathaniel Hudson, Hana Khamfroush, and Daniel E Lucani. 2020 · 2020
Cited alongside, same era.
Models and Processes to Extract Drug-like Molecules From Natural Language Text
Zhi Hong, J. Gregory Pauloski, Logan Ward, Kyle Chard, Ben Blaiszik, and Ian Foster. 2021 · 2021
Cited alongside, same era.
QoS-aware placement of deep learning services on the edge with multiple service implementations. In 2021 International Conference on Computer Communications and Networks (ICCCN) . IEEE, 1–8
Nathaniel Hudson, Hana Khamfroush, and Daniel E Lucani. 2021 · 2021
Cited alongside, same era.

Then

Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity
William Fedus, Barret Zoph, and Noam Shazeer. 2022 · 2022
Later among the works it cites.
Locating and editing factual associations in GPT
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022 · 2022
Later among the works it cites.
Galactica: A Large Language Model for Science
Original
Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. 2022 · 2022
Later among the works it cites.
LaMDA: Language Models for Dialog Applications
Original
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, et al · 2022
Later among the works it cites.
Interpretability in the wild: a circuit for indirect object identification in GPT-2 small
Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. 2022 · 2022
Later among the works it cites.
Alpa: Automating inter-and Intra-Operator parallelism for distributed deep learning. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22) . USENIX Association, 559–578
Lianmin Zheng, Zhuohan Li, Hao Zhang, Yonghao Zhuang, Zhifeng Chen, Yanping Huang, Yida Wang, et al · 2022
Later among the works it cites.
GenSLMs: Genome-scale language models reveal SARS-CoV-2 evolutionary dynamics
Original
Maxim Zvyagin, Alexander Brace, Kyle Hippe, Yuntian Deng, Bin Zhang, Cindy Orozco Bohorquez, Austin Clyde, et al · 2022
Later among the works it cites.
AWS SageMaker
Amazon Web Services. 2017 · 2023
Later among the works it cites.
Eliciting latent predictions from transformers with the tuned lens
Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. 2023 · 2023
Later among the works it cites.
Moformer: self-supervised transformer model for metal–organic framework property prediction
Zhonglin Cao, Rishikesh Magar, Yuyang Wang, and Amir Barati Farimani. 2023 · 2023
Later among the works it cites.
Dissecting recall of factual associations in auto-regressive language models
Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. 2023 · 2023
Later among the works it cites.
Why Meta’s latest large language model survived only three days online
Will Douglas Heaven. 2022 · 2023
Later among the works it cites.
Git-Theta: A Git Extension for Collaborative Development of Machine Learning Models
Nikhil Kandpal, Brian Lester, Mohammed Muqeeth, Anisha Mascarenhas, Monty Evans, Vishal Baskaran, Tenghao Huang, Haokun Liu, and Colin Raffel. 2023 · 2023
Later among the works it cites.
Progress measures for grokking via mechanistic interpretability
Neel Nanda, Lawrence Chan, Tom Liberum, Jess Smith, and Jacob Steinhardt. 2023 · 2023
Later among the works it cites.
Logit Lens on Non-GPT2 Models + Extensions
nostalgebraist. 2021 · 2023
Later among the works it cites.
Pinecone Vector Database
Pinecone. 2023 · 2023
Later among the works it cites.
PanGu- Σ \Sigma : Towards Trillion Parameter Language Model with Sparse Heterogeneous Computing
Original
Xiaozhe Ren, Pingyi Zhou, Xinfan Meng, Xinjing Huang, Yadao Wang, Weichao Wang, Pengfei Li, et al · 2023
Later among the works it cites.
Attention Lens: A Tool for Mechanistically Interpreting the Attention Head Information Retrieval Mechanism
Original
Mansi Sakarvadia, Arham Khan, Aswathy Ajith, Daniel Grzenda, Nathaniel Hudson, André Bauer, Kyle Chard, and Ian Foster. 2023b · 2023
Later among the works it cites.
Kubeflow
The Kubeflow Authors. 2018 · 2023
Later among the works it cites.
Cloud Services Enable Efficient AI-Guided Simulation Workflows across Heterogeneous Resources
Logan Ward, J. Gregory Pauloski, Valerie Hayot-Sasson, Ryan Chard, Yadu Babuji, Ganesh Sivaraman, et al · 2023
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…