Optimizing Deep Learning Recommender Systems' Training On CPU Cluster Architectures

M. Naumov, “On the dimensionality of embeddings for sparse features and data,” - , 2019. [Online]. Available: http://arxiv.org/abs/1901.02103

Original

1901

Earlier work this paper cites.

U. Gupta, X. Wang, M. Naumov, C. Wu, B. Reagen, D. Brooks, B. Cottel, K. M. Hazelwood, B. Jia, H. S. Lee, A. Malevich, D. Mudigere, M. Smelyanskiy, L. Xiong, and X. Zhang, “The architectural implications of facebook’s dnn-based personalized recommendation,” CoRR , vol. abs/1906.03109, 2019. [Online]. Available: https://arxiv.org/abs/1906.03109

Original

1906

Earlier work this paper cites.

A. Petitet, R. Whaley, J. Dongarra, and A. Cleary, “Hpl – a portable implementation of the high-performance linpack benchmark for distributed-memory computers,” - , 01 2008

2008

Earlier work this paper cites.

K. Goto and R. A. Geijn, “Anatomy of high-performance matrix multiplication,” ACM Transactions on Mathematical Software (TOMS) , vol. 34, no. 3, p. 12, 2008

2008

Earlier work this paper cites.

T. Hoefler and A. Lumsdaine, “Message progression in parallel computing - to thread or not to thread?” in 2008 IEEE International Conference on Cluster Computing , 2008, pp. 213–222

2008

Earlier work this paper cites.

V. Aggarwal, Y. Sabharwal, R. Garg, and P. Heidelberger, “Hpcc randomaccess benchmark for next generation supercomputers,” in 2009 IEEE International Symposium on Parallel Distributed Processing , May 2009, pp. 1–11

2009

Earlier work this paper cites.

“Download terabyte click logs,” 12/12/2013. [Online]. Available: https://labs.criteo.com/2013/12/download-terabyte-click-logs

2013

Earlier work this paper cites.

K. Vaidyanathan, K. Pamnany, D. D. Kalamkar, A. Heinecke, M. Smelyanskiy, J. Park, D. Kim, A. Shet, G, B. Kaul, B. Joo, and P. Dubey, “Improving communication performance and scalability of native applications on intel xeon phi coprocessor clusters,” in Proceedings of the 2014 IEEE 28th International Parallel and Distributed Processing Symposium , ser. IPDPS ’14. USA: IEEE Computer Society, 2014, p. 1083–1092. [Online]. Available: https://doi.org/10.1109/IPDPS.2014.113

2014

Earlier work this paper cites.

J. Dongarra, M. A. Heroux, and P. Luszczek, “A new metric for ranking high-performance computing systems,” National Science Review , vol. 3, no. 1, pp. 30–35, 01 2016. [Online]. Available: https://doi.org/10.1093/nsr/nwv084

2016

Earlier work this paper cites.

A. Breuer, A. Heinecke, and M. Bader, “Petascale local time stepping for the ader-dg finite element method,” in 2016 IEEE International Parallel and Distributed Processing Symposium (IPDPS) , 2016, pp. 854–863

2016

Earlier work this paper cites.

T. Akiba, S. Suzuki, and K. Fukuda, “Extremely large minibatch SGD: training resnet-50 on imagenet in 15 minutes,” CoRR , vol. abs/1711.04325, 2017. [Online]. Available: http://arxiv.org/abs/1711.04325

Original

2017

Earlier work this paper cites.

Y. You, Z. Zhang, C. Hsieh, and J. Demmel, “100-epoch imagenet training with alexnet in 24 minutes,” CoRR , vol. abs/1709.05011, 2017. [Online]. Available: http://arxiv.org/abs/1709.05011

Original

2017

Earlier work this paper cites.

N. P. Jouppi, C. Young, N. Patil, D. A. Patterson, G. Agrawal, R. Bajwa, S. Bates, S. Bhatia, N. Boden, A. Borchers, R. Boyle, P. Cantin, C. Chao, C. Clark, J. Coriell, M. Daley, M. Dau, J. Dean, B. Gelb, T. V. Ghaemmaghami, R. Gottipati, W. Gulland, R. Hagmann, R. C. Ho, D. Hogberg, J. Hu, R. Hundt, D. Hurt, J. Ibarz, A. Jaffey, A. Jaworski, A. Kaplan, H. Khaitan, A. Koch, N. Kumar, S. Lacy, J. Laudon, J. Law, D. Le, C. Leary, Z. Liu, K. Lucke, A. Lundin, G. MacKean, A. Maggiore, M. Mahony, K. Miller, R. Nagarajan, R. Narayanaswami, R. Ni, K. Nix, T. Norrie, M. Omernick, N. Penukonda, A. Phelps, J. Ross, A. Salek, E. Samadiani, C. Severn, G. Sizikov, M. Snelham, J. Souter, D. Steinberg, A. Swing, M. Tan, G. Thorson, B. Tian, H. Toma, E. Tuttle, V. Vasudevan, R. Walter, W. Wang, E. Wilcox, and D. H. Yoon, “In-datacenter performance analysis of a tensor processing unit,” CoRR , vol. abs/1704.04760, 2017. [Online]. Available: http://arxiv.org/abs/1704.04760

Original

2017

Earlier work this paper cites.

H. Mikami, H. Suganuma, P. U.-Chupala, Y. Tanaka, and Y. Kageyama, “Imagenet/resnet-50 training in 224 seconds,” CoRR , vol. abs/1811.05233, 2018. [Online]. Available: http://arxiv.org/abs/1811.05233

Original

2018

Earlier work this paper cites.

J. Zhang, J. Yang, and H. Yuen, “Training with low-precision embedding tables,” - , 2018

2018

Cited alongside, same era.

E. Georganas, S. Avancha, K. Banerjee, D. Kalamkar, G. Henry, H. Pabst, and A. Heinecke, “Anatomy of high-performance deep learning convolutions on simd architectures,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis , ser. SC ’18. Piscataway, NJ, USA: IEEE Press, 2018, pp. 66:1–66:12. [Online]. Available: https://doi.org/10.1109/SC.2018.00069

2018

Cited alongside, same era.

S. Sridharan, K. Vaidyanathan, D. D. Kalamkar, D. Das, M. E. Smorkalov, M. Shiryaev, D. Mudigere, N. Mellempudi, S. Avancha, B. Kaul, and P. Dubey, “On scale-out deep learning training for cloud and hpc,” ArXiv , vol. abs/1801.08030, 2018

Original

2018

Cited alongside, same era.

K. Hazelwood, S. Bird, D. Brooks, S. Chintala, U. Diril, D. Dzhulgakov, M. Fawzy, B. Jia, Y. Jia, A. Kalro, J. Law, K. Lee, J. Lu, P. Noordhuis, M. Smelyanskiy, L. Xiong, and X. Wang, “Applied machine learning at facebook: A datacenter infrastructure perspective,” in 2018 IEEE International Symposium on High Performance Computer Architecture (HPCA) , 2018, pp. 620–629

M. Naumov, D. Mudigere, H.-J. M. Shi, J. Huang, N. Sundaraman, J. Park, X. Wang, U. Gupta, C.-J. Wu, A. G. Azzolini, D. Dzhulgakov, A. Mallevich, I. Cherniavskii, Y. Lu, R. Krishnamoorthi, A. Yu, V. Y. Kondratenko, S. Pereira, X. Chen, W. Chen, V. Rao, B. Jia, L. Xiong, and M. Smelyanskiy, “Deep learning recommendation model for personalization and recommendation systems,” ArXiv , vol. abs/1906.00091, 2019

Original

2019

Later among the works it cites.

Facebook, “Dlrm,” https://github.com/facebookresearch/dlrm

2019

Later among the works it cites.

Intel, “oneccl,” https://github.com/oneapi-src/oneCCL

2019

Later among the works it cites.

“Facebook-optimized mlp on multisocket systems,” Accessed on 12/15/2019. [Online]. Available: https://github.com/jspark1105/tbb_test

2019

Later among the works it cites.

D. D. Kalamkar, D. Mudigere, N. Mellempudi, D. Das, K. Banerjee, S. Avancha, D. T. Vooturi, N. Jammalamadaka, J. Huang, H. Yuen, J. Yang, J. Park, A. Heinecke, E. Georganas, S. Srinivasan, A. Kundu, M. Smelyanskiy, B. Kaul, and P. Dubey, “A study of bfloat16 for deep learning training,” ArXiv , vol. abs/1905.12322, 2019