Built on

Li, Ang, Shuaiwen Leon Song, Jieyang Chen, Jiajia Li, Xu Liu, Nathan R. Tallent, and Kevin J. Barker. “Evaluating modern GPU interconnect: Pcie, nvlink, nv-sli, nvswitch and gpudirect.” IEEE Transactions on Parallel and Distributed Systems 31, no. 1 (2019): 94-110. See https://arxiv.org/abs/1903.04611
Original
1903
Earlier work this paper cites.
Gupta, Udit, Carole-Jean Wu, Xiaodong Wang, Maxim Naumov, Brandon Reagen, David Brooks, Bradford Cottel et al. “The architectural implications of facebook’s DNN-based personalized recommendation.” In 2020 IEEE International Symposium on High Performance Computer Architecture (HPCA), pp. 488-501. IEEE, 2020. See https://arxiv.org/abs/1906.03109
Original
1906
Earlier work this paper cites.
Naumov, Maxim, Dheevatsa Mudigere, Hao-Jun Michael Shi, Jianyu Huang, Narayanan Sundaraman, Jongsoo Park, Xiaodong Wang et al. “Deep learning recommendation model for personalization and recommendation systems.” arXiv preprint arXiv:1906.00091 (2019). See https://arxiv.org/abs/1906.00091
Original
1906
Earlier work this paper cites.
Reuther, Albert, Peter Michaleas, Michael Jones, Vijay Gadepally, Siddharth Samsi, and Jeremy Kepner. “Survey and benchmarking of machine learning accelerators.” arXiv preprint arXiv:1908.11348 (2019). See https://arxiv.org/abs/1908.11348
Original
1908
Earlier work this paper cites.
Jain, Paras, Ajay Jain, Aniruddha Nrusimha, Amir Gholami, Pieter Abbeel, Kurt Keutzer, Ion Stoica, and Joseph E. Gonzalez. “Checkmate: Breaking the memory wall with optimal tensor rematerialization.” arXiv preprint arXiv:1910.02653 (2019). See https://arxiv.org/abs/1910.02653
Original
1910
Earlier work this paper cites.
Rajbhandari, Samyam, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. “Zero: Memory optimization towards training a trillion parameter models.” arXiv preprint arXiv:1910.02054 (2019). https://arxiv.org/abs/1910.02054
Original
1910
Earlier work this paper cites.
Jia, Zhe, Blake Tillman, Marco Maggioni, and Daniele Paolo Scarpazza. “Dissecting the graphcore ipu architecture via microbenchmarking.” arXiv preprint arXiv:1912.03413 (2019). See https://arxiv.org/abs/1912.03413
Original
1912
Earlier work this paper cites.
Gupta, Udit, Samuel Hsia, Vikram Saraph, Xiaodong Wang, Brandon Reagen, Gu-Yeon Wei, Hsien-Hsin S. Lee, David Brooks, and Carole-Jean Wu. “DeepRecSys: A System for Optimizing End-To-End At-scale Neural Recommendation Inference.” arXiv preprint arXiv:2001.02772 (2020). See https://arxiv.org/abs/2001.02772
Original
2001
Earlier work this paper cites.
Naumov, Maxim, John Kim, Dheevatsa Mudigere, Srinivas Sridharan, Xiaodong Wang, Whitney Zhao, Serhat Yilmaz et al. “Deep Learning Training in Facebook Data Centers: Design of Scale-up and Scale-out Systems.” arXiv preprint arXiv:2003.09518 (2020). See https://arxiv.org/abs/2003.09518
Original
2003
Earlier work this paper cites.
Zhao, Weijie, Deping Xie, Ronglai Jia, Yulei Qian, Ruiquan Ding, Mingming Sun, and Ping Li. “Distributed Hierarchical GPU Parameter Server for Massive Scale Deep Learning Ads Systems.” arXiv preprint arXiv:2003.05622 (2020). See https://arxiv.org/abs/2003.05622
Original
2003
Earlier work this paper cites.
Kalamkar, Dhiraj, Evangelos Georganas, Sudarshan Srinivasan, Jianping Chen, Mikhail Shiryaev, and Alexander Heinecke. “Optimizing Deep Learning Recommender Systems Training On CPU Cluster Architectures.” arXiv preprint arXiv:2005.04680 (2020). See https://arxiv.org/abs/2005.04680
Original
2005
Earlier work this paper cites.
Hwang, Ranggi, Taehun Kim, Youngeun Kwon, and Minsoo Rhu. “Centaur: A Chiplet-based, Hybrid Sparse-Dense Accelerator for Personalized Recommendations.” arXiv preprint arXiv:2005.05968 (2020). See https://arxiv.org/abs/2005.05968
Original
2005
Earlier work this paper cites.
Chan, Ernie, Marcel Heimlich, Avi Purkayastha, and Robert Van De Geijn. “Collective communication: theory, practice, and experience.” Concurrency and Computation: Practice and Experience 19, no. 13 (2007): 1749-1783
2007
Earlier work this paper cites.
Varian, Hal R. “Online Ad auctions.” American Economic Review 99, no. 2 (2009): 430-34
2009
Earlier work this paper cites.
Rendle, Steffen. “Factorization machines.” In 2010 IEEE International Conference on Data Mining, pp. 995-1000. IEEE, 2010
2010
Earlier work this paper cites.

Similar

Graepel, Thore, Joaquin Quinonero Candela, Thomas Borchert, and Ralf Herbrich. “Web-scale bayesian click-through rate prediction for sponsored search advertising in microsoftś bing search engine.” Omnipress, 2010
2010
Cited alongside, same era.
He, Xinran, Junfeng Pan, Ou Jin, Tianbing Xu, Bo Liu, Tao Xu, Yanxin Shi et al. “Practical lessons from predicting clicks on ads at facebook.” In Proceedings of the Eighth International Workshop on Data Mining for Online Advertising, pp. 1-9. 2014
2014
Cited alongside, same era.
Chung, Junyoung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. “Empirical evaluation of gated recurrent neural networks on sequence modeling.” arXiv preprint arXiv:1412.3555 (2014)
Original
2014
Cited alongside, same era.
Li, Cheng, Yue Lu, Qiaozhu Mei, Dong Wang, and Sandeep Pandey. “Click-through prediction for advertising in twitter timeline.” In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1959-1968. 2015

Then

“Zion: Facebook Next-Generation Large Memory Training Platform”, Misha Smelyanskiy, Hot Chips 31, August 19, 2019. See https://www.hotchips.org/hc31/HC31_1.10_MethodologyAndMLSystem-Facebook-rev-d.pdf
2019
Later among the works it cites.
Zhao, Weijie, Jingyuan Zhang, Deping Xie, Yulei Qian, Ronglai Jia, and Ping Li. “AIBox: CTR prediction model training on a single node.” In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pp. 319-328. 2019. See http://research.baidu.com/Public/uploads/5e18a1017a7a0.pdf
2019
Later among the works it cites.
Kwon, Youngeun, Yunjae Lee, and Minsoo Rhu. “Tensordimm: A practical near-memory processing architecture for embeddings and tensor operations in deep learning.” In Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture, pp. 740-753. 2019
2019
Later among the works it cites.
Eitan Medina, Hot Chips 2019 presentation, Habana
2019
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2020

Accelerating Recommender Systems via Hardware "scale-in"

Krishna, Suresh, Krishna, Ravi

Understand

In today's era of "scale-out", this paper makes the case that a specialized hardware architecture based on "scale-in"--placing as many specialized processors as possible along with their memory systems and interconnect links within one or two boards in a rack--would offer the potential to boost large recommender system throughput by 12-62x for inference and 12-45x for training compared to the DGX-2 state-of-the-art AI platform, while minimizing the performance impact of distributing large models across multiple processors.

By analyzing Facebook's representative model--Deep Learning Recommendation Model (DLRM)--from a hardware architecture perspective, we quantify the impact on throughput of hardware parameters such as memory system design, collective communications latency and bandwidth, and interconnect topology.
By focusing on conditions that stress hardware, our analysis reveals limitations of existing AI accelerators and hardware platforms.

Built on

Li, Ang, Shuaiwen Leon Song, Jieyang Chen, Jiajia Li, Xu Liu, Nathan R. Tallent, and Kevin J. Barker. “Evaluating modern GPU interconnect: Pcie, nvlink, nv-sli, nvswitch and gpudirect.” IEEE Transactions on Parallel and Distributed Systems 31, no. 1 (2019): 94-110. See https://arxiv.org/abs/1903.04611
Original
1903
Earlier work this paper cites.
Gupta, Udit, Carole-Jean Wu, Xiaodong Wang, Maxim Naumov, Brandon Reagen, David Brooks, Bradford Cottel et al. “The architectural implications of facebook’s DNN-based personalized recommendation.” In 2020 IEEE International Symposium on High Performance Computer Architecture (HPCA), pp. 488-501. IEEE, 2020. See https://arxiv.org/abs/1906.03109
Original
1906
Earlier work this paper cites.
Naumov, Maxim, Dheevatsa Mudigere, Hao-Jun Michael Shi, Jianyu Huang, Narayanan Sundaraman, Jongsoo Park, Xiaodong Wang et al. “Deep learning recommendation model for personalization and recommendation systems.” arXiv preprint arXiv:1906.00091 (2019). See https://arxiv.org/abs/1906.00091
Original
1906
Earlier work this paper cites.
Reuther, Albert, Peter Michaleas, Michael Jones, Vijay Gadepally, Siddharth Samsi, and Jeremy Kepner. “Survey and benchmarking of machine learning accelerators.” arXiv preprint arXiv:1908.11348 (2019). See https://arxiv.org/abs/1908.11348
Original
1908
Earlier work this paper cites.
Jain, Paras, Ajay Jain, Aniruddha Nrusimha, Amir Gholami, Pieter Abbeel, Kurt Keutzer, Ion Stoica, and Joseph E. Gonzalez. “Checkmate: Breaking the memory wall with optimal tensor rematerialization.” arXiv preprint arXiv:1910.02653 (2019). See https://arxiv.org/abs/1910.02653
Original
1910
Earlier work this paper cites.
Rajbhandari, Samyam, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. “Zero: Memory optimization towards training a trillion parameter models.” arXiv preprint arXiv:1910.02054 (2019). https://arxiv.org/abs/1910.02054
Original
1910
Earlier work this paper cites.
Jia, Zhe, Blake Tillman, Marco Maggioni, and Daniele Paolo Scarpazza. “Dissecting the graphcore ipu architecture via microbenchmarking.” arXiv preprint arXiv:1912.03413 (2019). See https://arxiv.org/abs/1912.03413
Original
1912
Earlier work this paper cites.
Gupta, Udit, Samuel Hsia, Vikram Saraph, Xiaodong Wang, Brandon Reagen, Gu-Yeon Wei, Hsien-Hsin S. Lee, David Brooks, and Carole-Jean Wu. “DeepRecSys: A System for Optimizing End-To-End At-scale Neural Recommendation Inference.” arXiv preprint arXiv:2001.02772 (2020). See https://arxiv.org/abs/2001.02772
Original
2001
Earlier work this paper cites.
Naumov, Maxim, John Kim, Dheevatsa Mudigere, Srinivas Sridharan, Xiaodong Wang, Whitney Zhao, Serhat Yilmaz et al. “Deep Learning Training in Facebook Data Centers: Design of Scale-up and Scale-out Systems.” arXiv preprint arXiv:2003.09518 (2020). See https://arxiv.org/abs/2003.09518
Original
2003
Earlier work this paper cites.
Zhao, Weijie, Deping Xie, Ronglai Jia, Yulei Qian, Ruiquan Ding, Mingming Sun, and Ping Li. “Distributed Hierarchical GPU Parameter Server for Massive Scale Deep Learning Ads Systems.” arXiv preprint arXiv:2003.05622 (2020). See https://arxiv.org/abs/2003.05622
Original
2003
Earlier work this paper cites.
Kalamkar, Dhiraj, Evangelos Georganas, Sudarshan Srinivasan, Jianping Chen, Mikhail Shiryaev, and Alexander Heinecke. “Optimizing Deep Learning Recommender Systems Training On CPU Cluster Architectures.” arXiv preprint arXiv:2005.04680 (2020). See https://arxiv.org/abs/2005.04680
Original
2005
Earlier work this paper cites.
Hwang, Ranggi, Taehun Kim, Youngeun Kwon, and Minsoo Rhu. “Centaur: A Chiplet-based, Hybrid Sparse-Dense Accelerator for Personalized Recommendations.” arXiv preprint arXiv:2005.05968 (2020). See https://arxiv.org/abs/2005.05968
Original
2005
Earlier work this paper cites.
Chan, Ernie, Marcel Heimlich, Avi Purkayastha, and Robert Van De Geijn. “Collective communication: theory, practice, and experience.” Concurrency and Computation: Practice and Experience 19, no. 13 (2007): 1749-1783
2007
Earlier work this paper cites.
Varian, Hal R. “Online Ad auctions.” American Economic Review 99, no. 2 (2009): 430-34
2009
Earlier work this paper cites.
Rendle, Steffen. “Factorization machines.” In 2010 IEEE International Conference on Data Mining, pp. 995-1000. IEEE, 2010
2010
Earlier work this paper cites.

Similar

Graepel, Thore, Joaquin Quinonero Candela, Thomas Borchert, and Ralf Herbrich. “Web-scale bayesian click-through rate prediction for sponsored search advertising in microsoftś bing search engine.” Omnipress, 2010
2010
Cited alongside, same era.
He, Xinran, Junfeng Pan, Ou Jin, Tianbing Xu, Bo Liu, Tao Xu, Yanxin Shi et al. “Practical lessons from predicting clicks on ads at facebook.” In Proceedings of the Eighth International Workshop on Data Mining for Online Advertising, pp. 1-9. 2014
2014
Cited alongside, same era.
Chung, Junyoung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. “Empirical evaluation of gated recurrent neural networks on sequence modeling.” arXiv preprint arXiv:1412.3555 (2014)
Original
2014
Cited alongside, same era.
Li, Cheng, Yue Lu, Qiaozhu Mei, Dong Wang, and Sandeep Pandey. “Click-through prediction for advertising in twitter timeline.” In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1959-1968. 2015

Then

“Zion: Facebook Next-Generation Large Memory Training Platform”, Misha Smelyanskiy, Hot Chips 31, August 19, 2019. See https://www.hotchips.org/hc31/HC31_1.10_MethodologyAndMLSystem-Facebook-rev-d.pdf
2019
Later among the works it cites.
Zhao, Weijie, Jingyuan Zhang, Deping Xie, Yulei Qian, Ronglai Jia, and Ping Li. “AIBox: CTR prediction model training on a single node.” In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pp. 319-328. 2019. See http://research.baidu.com/Public/uploads/5e18a1017a7a0.pdf
2019
Later among the works it cites.
Kwon, Youngeun, Yunjae Lee, and Minsoo Rhu. “Tensordimm: A practical near-memory processing architecture for embeddings and tensor operations in deep learning.” In Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture, pp. 740-753. 2019
2019
Later among the works it cites.
Eitan Medina, Hot Chips 2019 presentation, Habana
2019
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

2015

Cited alongside, same era.

Kim, Yoongu, Weikun Yang, and Onur Mutlu. “Ramulator: A fast and extensible DRAM simulator.” IEEE Computer architecture letters 15, no. 1 (2015): 45-49

2015

Cited alongside, same era.

Gomez-Uribe, Carlos A., and Neil Hunt. “The netflix recommender system: Algorithms, business value, and innovation.” ACM Transactions on Management Information Systems (TMIS) 6, no. 4 (2015): 1-19

2015

Cited alongside, same era.

Arunkumar, Akhil, Evgeny Bolotin, Benjamin Cho, Ugljesa Milic, Eiman Ebrahimi, Oreste Villa, Aamer Jaleel, Carole-Jean Wu, and David Nellans. “MCM-GPU: Multi-chip-module GPUs for continued performance scalability.” ACM SIGARCH Computer Architecture News 45, no. 2 (2017): 320-332

2017

Cited alongside, same era.

Peng, Ivy Bo, Roberto Gioiosa, Gokcen Kestor, Pietro Cicotti, Erwin Laure, and Stefano Markidis. “Exploring the performance benefit of hybrid memory system on HPC environments.” In 2017 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), pp. 683-692. IEEE, 2017

2017

Cited alongside, same era.

Park, Jongsoo, Maxim Naumov, Protonu Basu, Summer Deng, Aravind Kalaiah, Daya Khudia, James Law et al. “Deep learning inference in facebook data centers: Characterization, performance optimizations and hardware implications.” arXiv preprint arXiv:1811.09886 (2018). See https://arxiv.org/abs/1811.09886

Original

2018

Cited alongside, same era.

Hazelwood, Kim, Sarah Bird, David Brooks, Soumith Chintala, Utku Diril, Dmytro Dzhulgakov, Mohamed Fawzy et al. “Applied machine learning at facebook: A datacenter infrastructure perspective.” In 2018 IEEE International Symposium on High Performance Computer Architecture (HPCA), pp. 620-629. IEEE, 2018

2018

Cited alongside, same era.

Hadash, Guy, Oren Sar Shalom, and Rita Osadchy. “Rank and rate: multi-task learning for recommender systems.”, In Proceedings of the 12th ACM Conference on Recommender Systems, pp. 451-454. 2018

2018

Cited alongside, same era.

Guo, Lin, Hui Ye, Wenbo Su, Henhuan Liu, Kai Sun, and Hang Xiang. “Visualizing and understanding deep neural networks in ctr prediction.” arXiv preprint arXiv:1806.08541 (2018)

Original

2018

Cited alongside, same era.

“The true Processing In Memory accelerator”, Fabrice Devaux, Hot Chips 31, August 19, 2019. See https://www.hotchips.org/hc31/HC31_1.4_UPMEM.FabriceDevaux.v2_1.pdf

2019