2022

Understanding Scaling Laws for Recommendation Models

Ardalani, Newsha, Wu, Carole-Jean, Chen, Zeliang et al.

Understand

Scale has been a major driving force in improving machine learning performance, and understanding scaling laws is essential for strategic planning for a sustainable model quality performance growth, long-term resource planning and developing efficient system infrastructures to support large-scale models.

In this paper, we study empirical scaling laws for DLRM style recommendation models, in particular Click-Through Rate (CTR).
We observe that model quality scales with power law plus constant in model size, data size and amount of compute used for training.
We characterize scaling efficiency along three different resource dimensions, namely data, parameters and compute by comparing the different scaling schemes along these axes.

Built on

Download Criteo 1TB Click Logs dataset
Criteo Labs · 2013
Earlier work this paper cites.
Kaggle Display Advertising Challenge Dataset
Criteo Labs · 2014
Earlier work this paper cites.
Practical lessons from predicting clicks on ads at facebook
He, X., Pan, J., Jin, O., Xu, T., Liu, B., Xu, T., Shi, Y., Atallah, A., Herbrich, R., Bowers, S., et al · 2014
Earlier work this paper cites.
A multi-view deep learning approach for cross domain user modeling in recommendation systems
Elkahky, A. M., Song, Y., and He, X · 2015
Earlier work this paper cites.
Deep neural networks for youtube recommendations
Covington, P., Adams, J., and Sargin, E · 2016
Earlier work this paper cites.
FAQ: All about the Google RankBrain algorithm
Sullivan, D · 2016
Earlier work this paper cites.
Deep learning scaling is predictable, empirically
Original
Hestness, J., Narang, S., Ardalani, N., Diamos, G., Jun, H., Kianinejad, H., Patwary, M., Ali, M., Yang, Y., and Zhou, Y · 2017
Earlier work this paper cites.
Related pins at pinterest: The evolution of a real-world recommender system
Liu, D. C., Rogers, S., Shiau, R., Kislyuk, D., Ma, K. C., Zhong, Z., Liu, J., and Jing, Y · 2017
Earlier work this paper cites.
Factorized deep retrieval and distributed tensorflow serving
Yi, X., Chen, Y.-F., Ramesh, S., Rajashekhar, V., Hong, L., Fiedel, N., Seshadri, N., Heldt, L., Wu, X., and Chi, E. H · 2018
Earlier work this paper cites.
Beyond human-level accuracy: Computational challenges in deep learning
Hestness, J., Ardalani, N., and Diamos, G · 2019
Earlier work this paper cites.
Deep learning recommendation model for personalization and recommendation systems
Original
Naumov, M., Mudigere, D., Shi, H.-J. M., Huang, J., Sundaraman, N., Park, J., Wang, X., Gupta, U., Wu, C.-J., Azzolini, A. G., et al · 2019
Earlier work this paper cites.

Similar

Aibox: Ctr prediction model training on a single node
Zhao, W., Zhang, J., Xie, D., Qian, Y., Jia, R., and Li, P · 2019
Cited alongside, same era.
Deep interest evolution network for click-through rate prediction
Zhou, G., Mou, N., Fan, Y., Pi, Q., Bian, W., Zhou, C., Zhu, X., and Gai, K · 2019
Cited alongside, same era.
Language models are few-shot learners
Original
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al · 2020
Cited alongside, same era.
Scaling laws for autoregressive generative modeling
Original
Henighan, T., Kaplan, J., Katz, M., Chen, M., Hesse, C., Jackson, J., Jun, H., Brown, T. B., Dhariwal, P., Gray, S., et al · 2020
Cited alongside, same era.
Scaling laws for neural language models
Original
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D · 2020
Cited alongside, same era.
Deep learning training in facebook data centers: Design of scale-up and scale-out systems
Original
Naumov, M., Kim, J., Mudigere, D., Sridharan, S., Wang, X., Zhao, W., Yilmaz, S., Kim, C., Yuen, H., Ozdal, M., et al · 2020
Cited alongside, same era.
Distributed hierarchical gpu parameter server for massive scale deep learning ads systems
Original
Zhao, W., Xie, D., Jia, R., Qian, Y., Ding, R., Sun, M., and Li, P · 2020
Cited alongside, same era.
Understanding training efficiency of deep learning recommendation models at scale
Acun, B., Murphy, M., Wang, X., Nie, J., Wu, C.-J., and Hazelwood, K · 2021
Cited alongside, same era.
Accelerating recommendation system training by leveraging popular choices
Adnan, M., Maboud, Y. E., Mahajan, D., and Nair, P. J · 2021
Cited alongside, same era.
Explaining neural scaling laws
Original
Bahri, Y., Dyer, E., Kaplan, J., Lee, J., and Sharma, U · 2021
Cited alongside, same era.
Data and parameter scaling laws for neural machine translation
Gordon, M. A., Duh, K., and Kaplan, J · 2021
Cited alongside, same era.

Then

Scaling laws for transfer
Original
Hernandez, D., Kaplan, J., Henighan, T., and McCandlish, S · 2021
Later among the works it cites.
Persia: A hybrid system scaling deep learning based recommenders up to 100 trillion parameters
Original
Lian, X., Yuan, B., Zhu, X., Wang, Y., He, Y., Wu, H., Sun, L., Lyu, H., Liu, C., Dong, X., et al · 2021
Later among the works it cites.
Understanding capacity-driven scale-out neural recommendation inference
Lui, M., Yetim, Y., Özkan, Ö., Zhao, Z., Tsai, S.-Y., Wu, C.-J., and Hempstead, M · 2021
Later among the works it cites.
Scaling laws for the few-shot adaptation of pre-trained image classifiers
Original
Prato, G., Guiroy, S., Caballero, E., Rish, I., and Chandar, S · 2021
Later among the works it cites.
Parameter, compute and data trends in machine learning
Sevilla, J., Villalobos, P., Cerón, J. F., Burtell, M., Heim, L., Nanjajjar, A. B., Ho, A., Besiroglu, T., Hobbhahn, M., and Denain, J.-S · 2021
Later among the works it cites.
Deep learning for recommender systems: A Netflix case study
Steck, H., Baltrunas, L., Elahi, E., Liang, D., Raimond, Y., and Basilico, J · 2021
Later among the works it cites.
Sustainable ai: Environmental implications, challenges and opportunities
Original
Wu, C.-J., Raghavendra, R., Gupta, U., Acun, B., Ardalani, N., Maeng, K., Chang, G., Behram, F. A., Huang, J., Bai, C., et al · 2021
Later among the works it cites.
Scaling vision transformers
Original
Zhai, X., Kolesnikov, A., Houlsby, N., and Beyer, L · 2021
Later among the works it cites.
Software-hardware co-design for fast and scalable training of deep learning recommendation models
Mudigere, D., Hao, Y., Huang, J., Jia, Z., Tulloch, A., Sridharan, S., Liu, X., Ozdal, M., Nie, J., Park, J., et al · 2022
Closest in time.
Beyond neural scaling laws: beating power law scaling via data pruning
Original
Sorscher, B., Geirhos, R., Shekhar, S., Ganguli, S., and Morcos, A. S · 2022
Closest in time.
Understanding data storage and ingestion for large-scale deep recommendation model training, 2022
Zhao, M., Agarwal, N., Basant, A., Gedik, B., Pan, S., Ozdal, M., Komuravelli, R., Pan, J., Bao, T., Lu, H., Narayanan, S., Langman, J., Wilfong, K., Rastogi, H., Wu, C.-J., Kozyrakis, C., and Pol, P · 2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…