2022

BagPipe: Accelerating Deep Recommendation Model Training

Agarwal, Saurabh, Yan, Chengpo, Zhang, Ziyi et al.

Understand

Deep learning based recommendation models (DLRM) are widely used in several business critical applications.

Training such recommendation models efficiently is challenging because they contain billions of embedding-based parameters, leading to significant overheads from embedding access.
By profiling existing systems for DLRM training, we observe that around 75\% of the iteration time is spent on embedding access and model synchronization.
Our key insight in this paper is that embedding access has a specific structure which can be used to accelerate training.

Built on

Selected studies of the principle of relative frequency in language
George Kingsley Zipf · 1932
Earlier work this paper cites.
A study of replacement algorithms for a virtual-storage computer
Laszlo A. Belady · 1966
Earlier work this paper cites.
Critical path analysis for the execution of parallel and distributed programs
C-Q Yang and Barton P Miller · 1988
Earlier work this paper cites.
Catastrophic forgetting in connectionist networks
Robert M French · 1999
Earlier work this paper cites.
Power-law distribution of the world wide web
Lada A Adamic and Bernardo A Huberman · 2000
Earlier work this paper cites.
Optimization of collective communication operations in mpich
Rajeev Thakur, Rolf Rabenseifner, and William Gropp · 2005
Earlier work this paper cites.
Community structure in large networks: Natural cluster sizes and the absence of large well-defined clusters
Jure Leskovec, Kevin J Lang, Anirban Dasgupta, and Michael W Mahoney · 2009
Earlier work this paper cites.
Deep content-based music recommendation
Aäron Van Den Oord, Sander Dieleman, and Benjamin Schrauwen · 2013
Earlier work this paper cites.
Adam: A method for stochastic optimization
Original
Diederik P Kingma and Jimmy Ba · 2014
Earlier work this paper cites.
Scaling distributed machine learning with the parameter server
Mu Li, David G Andersen, Jun Woo Park, Alexander J Smola, Amr Ahmed, Vanja Josifovski, James Long, Eugene J Shekita, and Bor-Yiing Su · 2014
Earlier work this paper cites.
Improving content-based and hybrid music recommendation using deep learning
Xinxi Wang and Ye Wang · 2014
Earlier work this paper cites.
Wide & deep learning for recommender systems
Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al · 2016
Earlier work this paper cites.
Deep neural networks for youtube recommendations
Paul Covington, Jay Adams, and Emre Sargin · 2016
Earlier work this paper cites.
The snowflake elastic data warehouse
Benoit Dageville, Thierry Cruanes, Marcin Zukowski, Vadim Antonov, Artin Avanes, Jon Bock, Jonathan Claybaugh, Daniel Engovatov, Martin Hentschel, Jiansheng Huang, et al · 2016
Earlier work this paper cites.
On deep learning for trust-aware recommendations in social networks
Shuiguang Deng, Longtao Huang, Guandong Xu, Xindong Wu, and Zhaohui Wu · 2016
Earlier work this paper cites.
What learning systems do intelligent agents need? complementary learning systems theory updated
Dharshan Kumaran, Demis Hassabis, and James L McClelland · 2016
Earlier work this paper cites.
Improved recurrent neural networks for session-based recommendations
Yong Kiam Tan, Xinxing Xu, and Yong Liu · 2016
Earlier work this paper cites.
Modelling contextual information in session-aware recommender systems with neural networks
Bartłomiej Twardowski · 2016
Earlier work this paper cites.
Deepfm: A factorization-machine based neural network for ctr prediction
Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He · 2017
Earlier work this paper cites.
Neural collaborative filtering
Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua · 2017
Earlier work this paper cites.
Related pins at pinterest: The evolution of a real-world recommender system
David C Liu, Stephanie Rogers, Raymond Shiau, Dmitry Kislyuk, Kevin C Ma, Zhigang Zhong, Jenny Liu, and Yushi Jing · 2017
Earlier work this paper cites.
Improving click-through rate prediction accuracy in online advertising by transfer learning
Yuhan Su, Zhongming Jin, Ying Chen, Xinghai Sun, Yaming Yang, Fangzheng Qiao, Fen Xia, and Wei Xu · 2017
Earlier work this paper cites.
3d convolutional networks for session-based recommendation with content features
Trinh Xuan Tuan and Tu Minh Phuong · 2017
Earlier work this paper cites.

Similar

Deep & cross network for ad click predictions
Ruoxi Wang, Bin Fu, Gang Fu, and Mingliang Wang · 2017
Cited alongside, same era.
What your images reveal: Exploiting visual contents for point-of-interest recommendation
Suhang Wang, Yilin Wang, Jiliang Tang, Kai Shu, Suhas Ranganath, and Huan Liu · 2017
Cited alongside, same era.
Collaborative deep metric learning for video understanding
Joonseok Lee, Sami Abu-El-Haija, Balakrishnan Varadarajan, and Apostol Natsev · 2018
Cited alongside, same era.
Learning from history and present: Next-item recommendation via discriminatively exploiting user behaviors
Zhi Li, Hongke Zhao, Qi Liu, Zhenya Huang, Tao Mei, and Enhong Chen · 2018
Cited alongside, same era.
Lectures on convex optimization
Yurii Nesterov et al · 2018
Cited alongside, same era.
Local sgd converges fast and communicates little
Original
Sebastian U Stich · 2018
Cited alongside, same era.
Visual background recommendation for dance performances using deep matrix factorization
Jiqing Wen, James She, Xiaopeng Li, and Hui Mao · 2018
Cited alongside, same era.
Gpipe: Efficient training of giant neural networks using pipeline parallelism
Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al · 2019
Cited alongside, same era.
Beyond data and model parallelism for deep neural networks
Zhihao Jia, Matei Zaharia, and Alex Aiken · 2019
Cited alongside, same era.
Pipedream: generalized pipeline parallelism for dnn training
Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R Devanur, Gregory R Ganger, Phillip B Gibbons, and Matei Zaharia · 2019
Cited alongside, same era.
Deep learning recommendation model for personalization and recommendation systems
Original
Maxim Naumov, Dheevatsa Mudigere, Hao-Jun Michael Shi, Jianyu Huang, Narayanan Sundaraman, Jongsoo Park, Xiaodong Wang, Udit Gupta, Carole-Jean Wu, Alisson G Azzolini, et al · 2019
Cited alongside, same era.
Pipeswitch: Fast pipelined context switching for deep learning applications
Zhihao Bai, Zhen Zhang, Yibo Zhu, and Xin Jin · 2020
Cited alongside, same era.

Then

P3: Distributed deep graph learning at scale
Swapnil Gandhi and Anand Padmanabha Iyer · 2021
Later among the works it cites.
Training recommender systems at scale: Communication-efficient model and data parallelism
Vipul Gupta, Dhruv Choudhary, Peter Tang, Xiaohan Wei, Xing Wang, Yuzhen Huang, Arun Kejariwal, Kannan Ramchandran, and Michael W Mahoney · 2021
Later among the works it cites.
Hierarchical training: Scaling deep recommendation models on large cpu clusters
Yuzhen Huang, Xiaohan Wei, Xing Wang, Jiyan Yang, Bor-Yiing Su, Shivam Bharuka, Dhruv Choudhary, Zewei Jiang, Hai Zheng, and Jack Langman · 2021
Later among the works it cites.
Oort: Efficient federated learning via guided participant selection
Fan Lai, Xiangfeng Zhu, Harsha V Madhyastha, and Mosharaf Chowdhury · 2021
Later among the works it cites.
Persia: A hybrid system scaling deep learning based recommenders up to 100 trillion parameters
Original
Xiangru Lian, Binhang Yuan, Xuefeng Zhu, Yulong Wang, Yongjun He, Honghuan Wu, Lei Sun, Haodong Lyu, Chengjun Liu, Xing Dong, et al · 2021
Later among the works it cites.
Marius: Learning massive graph embeddings on a single machine
Jason Mohoney, Roger Waleffe, Henry Xu, Theodoros Rekatsinas, and Shivaram Venkataraman · 2021
Later among the works it cites.
Software-hardware co-design for fast and scalable training of deep learning recommendation models
Original
Dheevatsa Mudigere, Yuchen Hao, Jianyu Huang, Zhihao Jia, Andrew Tulloch, Srinivas Sridharan, Xing Liu, Mustafa Ozdal, Jade Nie, Jongsoo Park, et al · 2021
Later among the works it cites.
Facebook research dlrm
Facebook Research · 2021
Later among the works it cites.
Switchflow: preemptive multitasking for deep learning
Xiaofeng Wu, Jia Rao, Wei Chen, Hang Huang, Chris Ding, and Heng Huang · 2021
Later among the works it cites.
Tt-rec: Tensor train compression for deep learning recommendation models
Chunxing Yin, Bilge Acun, Carole-Jean Wu, and Xing Liu · 2021
Later among the works it cites.
Understanding and co-designing the data ingestion pipeline for industry-scale recsys training
Original
Mark Zhao, Niket Agarwal, Aarti Basant, Bugra Gedik, Satadru Pan, Mustafa Ozdal, Rakesh Komuravelli, Jerry Pan, Tianshu Bao, Haowei Lu, et al · 2021
Later among the works it cites.
{ \{ Check-N-Run } \} : a checkpointing system for training deep learning recommendation models
Assaf Eisenman, Kiran Kumar Matam, Steven Ingram, Dheevatsa Mudigere, Raghuraman Krishnamoorthi, Krishnakumar Nair, Misha Smelyanskiy, and Murali Annavaram · 2022
Closest in time.
Training personalized recommendation systems from (gpu) scratch: look forward not backwards
Youngeun Kwon and Minsoo Rhu · 2022
Closest in time.
Monolith: real time recommendation system with collisionless embedding table
Original
Zhuoran Liu, Leqi Zou, Xuan Zou, Caihua Wang, Biao Zhang, Da Tang, Bolin Zhu, Yijie Zhu, Peng Wu, Ke Wang, et al · 2022
Closest in time.
Torchrec
Meta · 2022
Closest in time.
Het: Scaling out huge embedding model training via cache-enabled distributed framework
Xupeng Miao, Hailin Zhang, Yining Shi, Xiaonan Nie, Zhi Yang, Yangyu Tao, and Bin Cui · 2022
Closest in time.
https://bit.ly/341nGfs
Massively scale your deep learning training with nccl 2.4 · 2023
Closest in time.
Avazu click logs
Avazu Inc · 2023
Closest in time.
Criteo kaggle logs
Criteo Labs · 2023
Closest in time.
Terabyte click logs
Criteo Labs · 2023
Closest in time.
https://www.rookiehpc.com/mpi/docs/mpi_alltoall.php , 2019
Mpi all to all · 2023
Closest in time.
Facebook research dlrm
Facebook Research · 2023
Closest in time.
Fifo queues are all you need for cache eviction
Juncheng Yang, Yazhuo Zhang, Ziyue Qiu, Yao Yue, and Rashmi Vinayak · 2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…