mLoRA: Fine-Tuning LoRA Adapters via Highly-Efficient Pipeline Parallelism in Multiple GPUs

GPTuner: A Manual-Reading Database Tuning System via GPT-Guided Bayesian Optimization

Jiale Lao, Yibo Wang, Yufei Li, Jianping Wang, Yunjia Zhang, Zhiyuan Cheng, Wanghu Chen, Mingjie Tang, and Jianguo Wang. 2024 · 1952

Earlier work this paper cites.

Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning

Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. 2022 · 1965

Earlier work this paper cites.

A bridging model for parallel computation

Leslie G Valiant. 1990 · 1990

Earlier work this paper cites.

Pipelined Back-Propagation for Context-Dependent Deep Neural Networks.. In Interspeech . 26–29

Xie Chen, Adam Eversole, Gang Li, Dong Yu, and Frank Seide. 2012 · 2012

Earlier work this paper cites.

Efficient mini-batch training for stochastic optimization. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining . 661–670

Mu Li, Tong Zhang, Yuqiang Chen, and Alexander J Smola. 2014 · 2014

Earlier work this paper cites.

Training deep nets with sublinear memory cost

Original

Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. 2016 · 2016

Earlier work this paper cites.

Learning multiple visual domains with residual adapters

Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. 2017 · 2017

Earlier work this paper cites.

Efficient and robust parallel dnn training through model parallelism on multi-gpu platform

Original

Chi-Chung Chen, Chia-Lin Yang, and Hsiang-Yun Cheng. 2018b · 2018

Earlier work this paper cites.

Pipe-SGD: A decentralized pipelined SGD framework for distributed deep net training

Youjie Li, Mingchao Yu, Songze Li, Salman Avestimehr, Nam Sung Kim, and Alexander Schwing. 2018 · 2018

Earlier work this paper cites.

Asynchronous decentralized parallel stochastic gradient descent. In International Conference on Machine Learning . PMLR, 3043–3052

Xiangru Lian, Wei Zhang, Ce Zhang, and Ji Liu. 2018 · 2018

Earlier work this paper cites.

CUDA Graphs

2019 · 2019

Earlier work this paper cites.

Demystifying parallel and distributed deep learning: An in-depth concurrency analysis

Tal Ben-Nun and Torsten Hoefler. 2019 · 2019

Earlier work this paper cites.

Parameter-efficient transfer learning for NLP. In International conference on machine learning . PMLR, 2790–2799

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019 · 2019

Earlier work this paper cites.

Gpipe: Efficient training of giant neural networks using pipeline parallelism

Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al · 2019

Earlier work this paper cites.

Beyond Data and Model Parallelism for Deep Neural Networks

Zhihao Jia, Matei Zaharia, and Alex Aiken. 2019 · 2019

Earlier work this paper cites.

SwarmSGD: Scalable decentralized SGD with local updates

Original

Giorgi Nadiradze, Amirmojtaba Sabour, Dan Alistarh, Aditya Sharma, Ilia Markov, and Vitaly Aksenov. 2019 · 2019

Earlier work this paper cites.

PipeDream: generalized pipeline parallelism for DNN training. In Proceedings of the 27th ACM symposium on operating systems principles . 1–15

Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R Devanur, Gregory R Ganger, Phillip B Gibbons, and Matei Zaharia. 2019 · 2019

Earlier work this paper cites.

Pytorch: An imperative style, high-performance deep learning library

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al · 2019

Earlier work this paper cites.

Understanding the overheads of launching CUDA kernels

Lingqi Zhang, Mohamed Wahib, and Satoshi Matsuoka. 2019 · 2019

Earlier work this paper cites.

Parameter-efficient transfer learning with diff pruning

Original

Demi Guo, Alexander M Rush, and Yoon Kim. 2020 · 2020

Earlier work this paper cites.

Breaking (global) barriers in parallel stochastic optimization with wait-avoiding group averaging

Shigang Li, Tal Ben-Nun, Giorgi Nadiradze, Salvatore Di Girolamo, Nikoli Dryden, Dan Alistarh, and Torsten Hoefler. 2020 · 2020

Earlier work this paper cites.

Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis . IEEE, 1–16

Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020 · 2020

Earlier work this paper cites.

Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining . 3505–3506

Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020 · 2020

Earlier work this paper cites.

Masking as an efficient alternative to finetuning for pretrained language models

Original

Mengjie Zhao, Tao Lin, Fei Mi, Martin Jaggi, and Hinrich Schütze. 2020 · 2020

Earlier work this paper cites.

Composable sparse fine-tuning for cross-lingual transfer

Original

Alan Ansell, Edoardo Maria Ponti, Anna Korhonen, and Ivan Vulić. 2021 · 2021

Earlier work this paper cites.

On the opportunities and risks of foundation models

Original

Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al · 2021

Earlier work this paper cites.

Why dataset properties bound the scalability of parallel machine learning training algorithms

Daning Cheng, Shigang Li, Hanping Zhang, Fen Xia, and Yunquan Zhang. 2021 · 2021

Earlier work this paper cites.

Training Verifiers to Solve Math Word Problems

Original

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021 · 2021

Earlier work this paper cites.

DAPPLE: A pipelined data parallel approach for training large models. In Proceedings of the 26th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming . 431–445

Shiqing Fan, Yi Rong, Chen Meng, Zongyan Cao, Siyu Wang, Zhen Zheng, Chuan Wu, Guoping Long, Jun Yang, Lixue Xia, et al · 2021

Earlier work this paper cites.

The power of scale for parameter-efficient prompt tuning

Original

Brian Lester, Rami Al-Rfou, and Noah Constant. 2021a · 2021

Earlier work this paper cites.

The power of scale for parameter-efficient prompt tuning

Original

Brian Lester, Rami Al-Rfou, and Noah Constant. 2021b · 2021

Cited alongside, same era.

Prefix-tuning: Optimizing continuous prompts for generation

Original

Xiang Lisa Li and Percy Liang. 2021 · 2021

Cited alongside, same era.

P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks

Original

Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. 2021 · 2021

Cited alongside, same era.

{ \{ Zero-offload } \} : Democratizing { \{ billion-scale } \} model training. In 2021 USENIX Annual Technical Conference (USENIX ATC 21) . 551–564

Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, and Yuxiong He. 2021 · 2021

Cited alongside, same era.

Llama: Open and efficient foundation language models

Original

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al · 2023

Closest in time.

Llama 2: Open foundation and fine-tuned chat models

Original

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al · 2023

Closest in time.

Hyperparameter optimization for Large Language Model instruction-tuning

C Tribes, S Benarroch-Lelong, P Lu, and I Kobyzev. 2023 · 2023

Closest in time.

Fast Distributed Inference Serving for Large Language Models

Original

Bingyang Wu, Yinmin Zhong, Zili Zhang, Gang Huang, Xuanzhe Liu, and Xin Jin. 2023 · 2023

Closest in time.

pycorrector: Text Error Correction Tool

Ming Xu. 2023 · 2023