2021

Persia: An Open, Hybrid System Scaling Deep Learning-based Recommenders up to 100 Trillion Parameters

Lian, Xiangru, Yuan, Binhang, Zhu, Xuefeng et al.

Understand

Deep learning based models have dominated the current landscape of production recommender systems.

Furthermore, recent years have witnessed an exponential growth of the model scale--from Google's 2016 model with 1 billion parameters to the latest Facebook's model with 12 trillion parameters.
Significant quality boost has come with each jump of the model capacity, which makes us believe the era of 100 trillion parameters is around the corner.
However, the training of such models is challenging even within industrial scale data centers.

Built on

Tag-aware personalized recommendation using a deep-semantic similarity model with negative sampling. In
Zhenghua Xu, Cheng Chen, Thomas Lukasiewicz, Yishu Miao, and Xiangwu Meng. 2016 · 1924
Earlier work this paper cites.
The YouTube video recommendation system. In
James Davidson, Benjamin Liebald, Junning Liu, Palash Nandy, Taylor Van Vleet, Ullas Gargi, Sujoy Gupta, Yu He, Mike Lambert, Blake Livingston, et al · 2010
Earlier work this paper cites.
HOGWILD! a lock-free approach to parallelizing stochastic gradient descent. In
Feng Niu, Benjamin Recht, Christopher Re, and Stephen J Wright. 2011 · 2011
Earlier work this paper cites.
Large scale distributed deep networks. In
Jeffrey Dean, Greg S Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Quoc V Le, Mark Z Mao, Marc’Aurelio Ranzato, Andrew Senior, Paul Tucker, et al · 2012
Earlier work this paper cites.
Deep content-based music recommendation. In
Aäron Van Den Oord, Sander Dieleman, and Benjamin Schrauwen. 2013 · 2013
Earlier work this paper cites.
Adam: A method for stochastic optimization
Original
Diederik P Kingma and Jimmy Ba. 2014 · 2014
Earlier work this paper cites.
Scaling distributed machine learning with the parameter server. In
Mu Li, David G Andersen, Jun Woo Park, Alexander J Smola, Amr Ahmed, Vanja Josifovski, James Long, Eugene J Shekita, and Bor-Yiing Su. 2014 · 2014
Earlier work this paper cites.
Improving content-based and hybrid music recommendation using deep learning. In
Xinxi Wang and Ye Wang. 2014 · 2014
Earlier work this paper cites.
Mxnet: A flexible and efficient machine learning library for heterogeneous distributed systems
Original
Tianqi Chen, Mu Li, Yutian Li, Min Lin, Naiyan Wang, Minjie Wang, Tianjun Xiao, Bing Xu, Chiyuan Zhang, and Zheng Zhang. 2015 · 2015
Earlier work this paper cites.
The netflix recommender system: Algorithms, business value, and innovation
Carlos A Gomez-Uribe and Neil Hunt. 2015 · 2015
Earlier work this paper cites.
Asynchronous parallel stochastic gradient for nonconvex optimization
Xiangru Lian, Yijun Huang, Yuncheng Li, and Ji Liu. 2015 · 2015
Earlier work this paper cites.
Autorec: Autoencoders meet collaborative filtering. In
Suvash Sedhain, Aditya Krishna Menon, Scott Sanner, and Lexing Xie. 2015 · 2015
Earlier work this paper cites.
Tensorflow: A system for large-scale machine learning. In
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al · 2016
Earlier work this paper cites.
QSGD: Communication-efficient SGD via gradient quantization and encoding
Original
Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovic. 2016 · 2016
Earlier work this paper cites.
Revisiting distributed synchronous SGD
Original
Jianmin Chen, Xinghao Pan, Rajat Monga, Samy Bengio, and Rafal Jozefowicz. 2016 · 2016
Earlier work this paper cites.
Xgboost: A scalable tree boosting system. In
Tianqi Chen and Carlos Guestrin. 2016 · 2016
Earlier work this paper cites.
Wide & deep learning for recommender systems. In
Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al · 2016
Earlier work this paper cites.
Deep neural networks for youtube recommendations. In
Paul Covington, Jay Adams, and Emre Sargin. 2016 · 2016
Earlier work this paper cites.
On deep learning for trust-aware recommendations in social networks
Shuiguang Deng, Longtao Huang, Guandong Xu, Xindong Wu, and Zhaohui Wu. 2016 · 2016
Earlier work this paper cites.
Immersive recommendation: News and event recommendations using personal digital traces. In
Cheng-Kang Hsieh, Longqi Yang, Honghao Wei, Mor Naaman, and Deborah Estrin. 2016 · 2016
Earlier work this paper cites.
A Comprehensive Linear Speedup Analysis for Asynchronous Stochastic Parallel Optimization from Zeroth-Order to First-Order
Xiangru Lian, Huan Zhang, Cho-Jui Hsieh, Yijun Huang, and Ji Liu. 2016 · 2016
Earlier work this paper cites.
Improved recurrent neural networks for session-based recommendations. In
Yong Kiam Tan, Xinxing Xu, and Yong Liu. 2016 · 2016
Earlier work this paper cites.
Modelling contextual information in session-aware recommender systems with neural networks. In
Bartłomiej Twardowski. 2016 · 2016
Earlier work this paper cites.
Multi-objective optimization for long tail recommendation
Shanfeng Wang, Maoguo Gong, Haoliang Li, and Junwei Yang. 2016 · 2016
Earlier work this paper cites.
Recurrent neural network based recommendation for time heterogeneous feedback
Caihua Wu, Junwei Wang, Juntao Liu, and Wenyu Liu. 2016b · 2016
Earlier work this paper cites.
Personal recommendation using deep recurrent neural networks in NetEase. In
Sai Wu, Weichao Ren, Chengchao Yu, Gang Chen, Dongxiang Zhang, and Jingbo Zhu. 2016a · 2016
Earlier work this paper cites.
Tag-aware recommender systems based on deep neural networks
Yi Zuo, Jiulin Zeng, Maoguo Gong, and Licheng Jiao. 2016 · 2016
Earlier work this paper cites.
Attentive collaborative filtering: Multimedia recommendation with item-and component-level attention. In
Jingyuan Chen, Hanwang Zhang, Xiangnan He, Liqiang Nie, Wei Liu, and Tat-Seng Chua. 2017 · 2017
Earlier work this paper cites.
Heterogeneity-aware distributed parameter servers. In
Jiawei Jiang, Bin Cui, Ce Zhang, and Lele Yu. 2017 · 2017
Earlier work this paper cites.
Can decentralized algorithms outperform centralized algorithms? a case study for decentralized parallel stochastic gradient descent. In
Xiangru Lian, Ce Zhang, Huan Zhang, Cho-Jui Hsieh, Wei Zhang, and Ji Liu. 2017 · 2017
Earlier work this paper cites.
Related pins at pinterest: The evolution of a real-world recommender system. In
David C Liu, Stephanie Rogers, Raymond Shiau, Dmitry Kislyuk, Kevin C Ma, Zhigang Zhong, Jenny Liu, and Yushi Jing. 2017 · 2017
Earlier work this paper cites.
Asynchronous distributed variational Gaussian process for regression. In
Hao Peng, Shandian Zhe, Xiao Zhang, and Yuan Qi. 2017 · 2017
Earlier work this paper cites.
Interpretable convolutional neural networks with dual local and global attention for review rating prediction. In
Sungyong Seo, Jing Huang, Hao Yang, and Yan Liu. 2017 · 2017
Earlier work this paper cites.
Improving click-through rate prediction accuracy in online advertising by transfer learning. In
Yuhan Su, Zhongming Jin, Ying Chen, Xinghai Sun, Yaming Yang, Fangzheng Qiao, Fen Xia, and Wei Xu. 2017 · 2017
Earlier work this paper cites.

Similar

3D convolutional networks for session-based recommendation with content features. In
Trinh Xuan Tuan and Tu Minh Phuong. 2017 · 2017
Cited alongside, same era.
TernGrad: ternary gradients to reduce communication in distributed deep learning. In
Wei Wen, Cong Xu, Feng Yan, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. 2017 · 2017
Cited alongside, same era.
Bridging collaborative filtering and semi-supervised learning: a neural approach for poi recommendation. In
Carl Yang, Lanxiao Bai, Chao Zhang, Quan Yuan, and Jiawei Han. 2017 · 2017
Cited alongside, same era.
Joint representation learning for top-n recommendation with heterogeneous information sources. In
Yongfeng Zhang, Qingyao Ai, Xu Chen, and W Bruce Croft. 2017a · 2017
Cited alongside, same era.
Asynchronous stochastic gradient descent with delay compensation. In
Shuxin Zheng, Qi Meng, Taifeng Wang, Wei Chen, Nenghai Yu, Zhi-Ming Ma, and Tie-Yan Liu. 2017 · 2017
Cited alongside, same era.
Representation learning via Dual-Autoencoder for recommendation
Fuzhen Zhuang, Zhiqiang Zhang, Mingda Qian, Chuan Shi, Xing Xie, and Qing He. 2017 · 2017
Cited alongside, same era.
The convergence of sparsified gradient methods. In
Dan Alistarh, Torsten Hoefler, Mikael Johansson, Sarit Khirirat, Nikola Konstantinov, and Cédric Renggli. 2018 · 2018
Cited alongside, same era.
signSGD: Compressed optimisation for non-convex problems. In
Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Animashree Anandkumar. 2018 · 2018
Cited alongside, same era.
Pixie: A system for recommending 3+ billion items to 200+ million users in real-time. In
Chantat Eksombatchai, Pranav Jindal, Jerry Zitao Liu, Yuchen Liu, Rahul Sharma, Charles Sugnet, Mark Ulrich, and Jure Leskovec. 2018 · 2018
Cited alongside, same era.
Pipedream: Fast and efficient pipeline parallel dnn training
Original
Aaron Harlap, Deepak Narayanan, Amar Phanishayee, Vivek Seshadri, Nikhil Devanur, Greg Ganger, and Phil Gibbons. 2018 · 2018
Cited alongside, same era.
Flexps: Flexible parallelism control in parameter server architecture
Yuzhen Huang, Tatiana Jin, Yidi Wu, Zhenkun Cai, Xiao Yan, Fan Yang, Jinfeng Li, Yuying Guo, and James Cheng. 2018 · 2018
Cited alongside, same era.
Dimboost: Boosting gradient boosting decision tree to higher dimensions. In
Jiawei Jiang, Bin Cui, Ce Zhang, and Fangcheng Fu. 2018 · 2018
Cited alongside, same era.

Then

Don’t Use Large Mini-batches, Use Local SGD. In
Tao Lin, Sebastian U Stich, Kumar Kshitij Patel, and Martin Jaggi. 2019 · 2019
Later among the works it cites.
Scaling multinomial logistic regression via hybrid parallelism. In
Parameswaran Raman, Sriram Srinivasan, Shin Matsushima, Xinhua Zhang, Hyokun Yun, and SVN Vishwanathan. 2019 · 2019
Later among the works it cites.
Megatron-lm: Training multi-billion parameter language models using model parallelism
Original
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019 · 2019
Later among the works it cites.
Deepsqueeze: Parallel stochastic gradient descent with double-pass error-compensated compression
Original
Hanlin Tang, Xiangru Lian, Shuang Qiu, Lei Yuan, Ce Zhang, Tong Zhang, and Ji Liu. 2019a · 2019
Later among the works it cites.
Adaptive Communication Strategies to Achieve the Best Error-Runtime Trade-off in Local-update SGD. In
Jianyu Wang and Gauri Joshi. 2019 · 2019
Later among the works it cites.
Supporting very large models using automatic dataflow graph partitioning. In
Minjie Wang, Chien-chin Huang, and Jinyang Li. 2019 · 2019
Later among the works it cites.
Deep learning based recommender system: A survey and new perspectives
Shuai Zhang, Lina Yao, Aixin Sun, and Yi Tay. 2019c · 2019
Later among the works it cites.
Ps2: Parameter server on spark. In
Zhipeng Zhang, Bin Cui, Yingxia Shao, Lele Yu, Jiawei Jiang, and Xupeng Miao. 2019a · 2019
Later among the works it cites.
Mllib*: Fast training of glms using spark mllib. In
Zhipeng Zhang, Jiawei Jiang, Wentao Wu, Ce Zhang, Lele Yu, and Bin Cui. 2019b · 2019
Later among the works it cites.
Deep interest evolution network for click-through rate prediction. In
Guorui Zhou, Na Mou, Ying Fan, Qi Pi, Weijie Bian, Chang Zhou, Xiaoqiang Zhu, and Kun Gai. 2019 · 2019
Later among the works it cites.
Understanding Training Efficiency of Deep Learning Recommendation Models at Scale
Original
Bilge Acun, Matthew Murphy, Xiaodong Wang, Jade Nie, Carole-Jean Wu, and Kim Hazelwood. 2020 · 2020
Later among the works it cites.
On biased compression for distributed learning
Original
Aleksandr Beznosikov, Samuel Horváth, Peter Richtárik, and Mher Safaryan. 2020 · 2020
Later among the works it cites.
A Unified Architecture for Accelerating Distributed
Yimin Jiang, Yibo Zhu, Chang Lan, Bairen Yi, Yong Cui, and Chuanxiong Guo. 2020 · 2020
Later among the works it cites.
Accelerating Recommender Systems via Hardware" scale-in"
Original
Suresh Krishna and Ravi Krishna. 2020 · 2020
Later among the works it cites.
Improving Multi-Scenario Learning to Rank in E-commerce by Exploiting Task Relationships in the Label Space. In
Pengcheng Li, Runze Li, Qing Da, An-Xiang Zeng, and Lijun Zhang. 2020 · 2020
Later among the works it cites.
Distributed Learning Systems with First-Order Methods
Ji Liu and Ce Zhang. 2020 · 2020
Later among the works it cites.
Deep learning training in facebook data centers: Design of scale-up and scale-out systems
Original
Maxim Naumov, John Kim, Dheevatsa Mudigere, Srinivas Sridharan, Xiaodong Wang, Whitney Zhao, Serhat Yilmaz, Changkyu Kim, Hector Yuen, Mustafa Ozdal, et al · 2020
Later among the works it cites.
Dissecting the Communication Latency in Distributed Deep Sparse Learning. In
Heng Pan, Zhenyu Li, JianBo Dong, Zheng Cao, Tao Lan, Di Zhang, Gareth Tyson, and Gaogang Xie. 2020 · 2020
Later among the works it cites.
HetPipe: Enabling Large
Jay H Park, Gyeongchan Yun, M Yi Chang, Nguyen T Nguyen, Seungmin Lee, Jaesik Choi, Sam H Noh, and Young-ri Choi. 2020 · 2020
Later among the works it cites.
Zero: Memory optimizations toward training trillion parameter models. In
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020 · 2020
Later among the works it cites.
Efficient Algorithms for Device Placement of DNN Graph Operators
Jakub M Tarnawski, Amar Phanishayee, Nikhil Devanur, Divya Mahajan, and Fanny Nina Paravecino. 2020 · 2020
Later among the works it cites.
Kraken: memory-efficient continual learning for large-scale real-time recommendations. In
Minhui Xie, Kai Ren, Youyou Lu, Guangxu Yang, Qingxing Xu, Bihai Wu, Jiazhen Lin, Hongbo Ao, Wanhong Xu, and Jiwu Shu. 2020 · 2020
Later among the works it cites.
Model Size Reduction Using Frequency Based Double Hashing for Recommender Systems. In
Caojin Zhang, Yicun Liu, Yuanpu Xie, Sofia Ira Ktena, Alykhan Tejani, Akshay Gupta, Pranay Kumar Myana, Deepak Dilipkumar, Suvadip Paul, Ikuhiro Ihara, et al · 2020
Later among the works it cites.
Distributed hierarchical gpu parameter server for massive scale deep learning ads systems
Original
Weijie Zhao, Deping Xie, Ronglai Jia, Yulei Qian, Ruiquan Ding, Mingming Sun, and Ping Li. 2020b · 2020
Later among the works it cites.
Autoemb: Automated embedding dimensionality search in streaming recommendations
Original
Xiangyu Zhao, Chong Wang, Ming Chen, Xudong Zheng, Xiaobing Liu, and Jiliang Tang. 2020a · 2020
Later among the works it cites.
Extreme multi-label learning for semantic matching in product search. In
Wei-Cheng Chang, Daniel Jiang, Hsiang-Fu Yu, Choon Hui Teo, Jiong Zhang, Kai Zhong, Kedarnath Kolluri, Qie Hu, Nikhil Shandilya, Vyacheslav Ievgrafov, et al · 2021
Closest in time.
BAGUA: Scaling up Distributed Learning with System Relaxations
Original
Shaoduo Gan, Xiangru Lian, Rui Wang, Jianbin Chang, Chengjun Liu, Hongmei Shi, Shengzhuo Zhang, Xianghong Li, Tengxu Sun, Jiawei Jiang, et al · 2021
Closest in time.
Amazon SageMaker Model Parallelism: A General and Flexible Framework for Large Model Training
Original
Can Karakus, Rahul Huilgol, Fei Wu, Anirudh Subramanian, Cade Daniel, Derya Cavdar, Teng Xu, Haohan Chen, Arash Rahnama, and Luis Quintela. 2021 · 2021
Closest in time.
TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Models
Original
Zhuohan Li, Siyuan Zhuang, Shiyuan Guo, Danyang Zhuo, Hao Zhang, Dawn Song, and Ion Stoica. 2021 · 2021
Closest in time.
Persia: A Hybrid System Scaling Deep Learning Based Recommenders up to 100 Trillion Parameters
Original
Xiangru Lian, Binhang Yuan, Xuefeng Zhu, Yulong Wang, Yongjun He, Honghuan Wu, Lei Sun, Haodong Lyu, Chengjun Liu, Xing Dong, et al · 2021
Closest in time.
High-performance, distributed training of large-scale deep learning recommendation models
Original
Dheevatsa Mudigere, Yuchen Hao, Jianyu Huang, Andrew Tulloch, Srinivas Sridharan, Xing Liu, Mustafa Ozdal, Jade Nie, Jongsoo Park, Liang Luo, et al · 2021
Closest in time.
RecSSD: Near Data Processing for Solid State Drive Based Recommendation Inference
Original
Mark Wilkening, Udit Gupta, Samuel Hsia, Caroline Trippel, Carole-Jean Wu, David Brooks, and Gu-Yeon Wei. 2021 · 2021
Closest in time.
HET: Scaling out Huge Embedding Model Training via Cache-enabled Distributed Framework
Xupeng Miao, Hailin Zhang, Yining Shi, Xiaonan Nie, Zhi Yang, Yangyu Tao, and Bin Cui. 2022 · 2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…