Distillation Matters: Empowering Sequential Recommenders to Match the Performance of Large Language Model

Language models are few-shot learners

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al · 1901

Earlier work this paper cites.

Large Language Models are Learnable Planners for Long-Term Recommendation. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval . 1893–1903

Wentao Shi, Xiangnan He, Yang Zhang, Chongming Gao, Xinyue Li, Jizhi Zhang, Qifan Wang, and Fuli Feng. 2024 · 1903

Earlier work this paper cites.

Accurate sum and dot product

Takeshi Ogita, Siegfried M Rump, and Shin’ichi Oishi. 2005 · 1988

Earlier work this paper cites.

Approximation theory of the MLP model in neural networks

Allan Pinkus. 1999 · 1999

Earlier work this paper cites.

Adam: A method for stochastic optimization

Original

Diederik P Kingma and Jimmy Ba. 2014 · 2014

Earlier work this paper cites.

Fitnets: Hints for thin deep nets

Romero Adriana, Ballas Nicolas, K Samira Ebrahimi, Chassang Antoine, Gatta Carlo, and Bengio Yoshua. 2015 · 2015

Earlier work this paper cites.

Session-based recommendations with recurrent neural networks

Original

Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. 2015 · 2015

Earlier work this paper cites.

Distilling the knowledge in a neural network

Original

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015 · 2015

Earlier work this paper cites.

Learning from multiple teacher networks. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining . 1285–1294

Shan You, Chang Xu, Chao Xu, and Dacheng Tao. 2017 · 2017

Earlier work this paper cites.

Sequential recommendation with user memory networks. In Proceedings of the eleventh ACM international conference on web search and data mining . 108–116

Xu Chen, Hongteng Xu, Yongfeng Zhang, Jiaxi Tang, Yixin Cao, Zheng Qin, and Hongyuan Zha. 2018 · 2018

Earlier work this paper cites.

Bert: Pre-training of deep bidirectional transformers for language understanding

Original

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018 · 2018

Earlier work this paper cites.

Self-attentive sequential recommendation. In 2018 IEEE international conference on data mining (ICDM) . IEEE, 197–206

Wang-Cheng Kang and Julian McAuley. 2018 · 2018

Earlier work this paper cites.

On the efficacy of knowledge distillation. In Proceedings of the IEEE/CVF international conference on computer vision . 4794–4802

Jang Hyun Cho and Bharath Hariharan. 2019 · 2019

Earlier work this paper cites.

Collaborative distillation for top-N recommendation. In 2019 IEEE International Conference on Data Mining (ICDM) . IEEE, 369–378

Jae-woong Lee, Minjin Choi, Jongwuk Lee, and Hyunjung Shim. 2019 · 2019

Earlier work this paper cites.

BERT4Rec: Sequential recommendation with bidirectional encoder representations from transformer. In Proceedings of the 28th ACM international conference on information and knowledge management . 1441–1450

Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. 2019 · 2019

Earlier work this paper cites.

Sequential recommender systems: challenges, progress and prospects

Original

Shoujin Wang, Liang Hu, Yan Wang, Longbing Cao, Quan Z Sheng, and Mehmet Orgun. 2019 · 2019

Earlier work this paper cites.

Agree to disagree: Adaptive ensemble knowledge distillation in gradient space

Shangchen Du, Shan You, Xiaojie Li, Jianlong Wu, Fei Wang, Chen Qian, and Changshui Zhang. 2020 · 2020

Earlier work this paper cites.

Deep learning for sequential recommendation: Algorithms, influential factors, and evaluations

Hui Fang, Danning Zhang, Yiheng Shu, and Guibing Guo. 2020 · 2020

Earlier work this paper cites.

Lightgcn: Simplifying and powering graph convolution network for recommendation. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval . 639–648

Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yongdong Zhang, and Meng Wang. 2020 · 2020

Earlier work this paper cites.

DE-RRD: A knowledge distillation framework for recommender system. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management . 605–614

SeongKu Kang, Junyoung Hwang, Wonbin Kweon, and Hwanjo Yu. 2020 · 2020

Earlier work this paper cites.

Time interval aware self-attention for sequential recommendation. In Proceedings of the 13th international conference on web search and data mining . 322–330

Jiacheng Li, Yujie Wang, and Julian McAuley. 2020 · 2020

Earlier work this paper cites.

Binary cross entropy with deep learning technique for image classification

Usha Ruby and Vamsidhar Yendapalli. 2020 · 2020

Earlier work this paper cites.

Ensembled CTR prediction via knowledge distillation. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management . 2941–2958

Jieming Zhu, Jinyang Liu, Weiqi Li, Jincai Lai, Xiuqiang He, Liang Chen, and Zibin Zheng. 2020 · 2020

Earlier work this paper cites.

Sequential recommendation with graph neural networks. In Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval . 378–387

Jianxin Chang, Chen Gao, Yu Zheng, Yiqun Hui, Yanan Niu, Yang Song, Depeng Jin, and Yong Li. 2021 · 2021

Earlier work this paper cites.

AutoDebias: Learning to debias for recommendation. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval . 21–30

Jiawei Chen, Hande Dong, Yang Qiu, Xiangnan He, Xin Xin, Liang Chen, Guli Lin, and Keping Yang. 2021 · 2021

Earlier work this paper cites.

Knowledge distillation: A survey

Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. 2021 · 2021

Cited alongside, same era.

Topology distillation for recommender system. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining . 829–839

SeongKu Kang, Junyoung Hwang, Wonbin Kweon, and Hwanjo Yu. 2021 · 2021

Cited alongside, same era.

Dual correction strategy for ranking distillation in top-n recommender system. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management . 3186–3190

Youngjune Lee and Kee-Eung Kim. 2021 · 2021

Cited alongside, same era.

Towards universal sequence representation learning for recommender systems. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining . 585–593

Yupeng Hou, Shanlei Mu, Wayne Xin Zhao, Yaliang Li, Bolin Ding, and Ji-Rong Wen. 2022 · 2022

Cited alongside, same era.

Knowledge distillation from a stronger teacher

Tao Huang, Shan You, Fei Wang, Chen Qian, and Chang Xu. 2022 · 2022

SIGformer: Sign-aware Graph Transformer for Recommendation. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval . 1274–1284

Sirui Chen, Jiawei Chen, Sheng Zhou, Bohao Wang, Shen Han, Chanfei Su, Yuqing Yuan, and Can Wang. 2024 · 2024

Closest in time.

Breaking the length barrier: Llm-enhanced CTR prediction in long textual user behaviors. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval . 2311–2315

Binzong Geng, Zhaoxin Huan, Xiaolu Zhang, Yong He, Liang Zhang, Fajie Yuan, Jun Zhou, and Linjian Mo. 2024 · 2024

Closest in time.

Large language models are zero-shot rankers for recommender systems. In European Conference on Information Retrieval . Springer, 364–381

Yupeng Hou, Junjie Zhang, Zihan Lin, Hongyu Lu, Ruobing Xie, Julian McAuley, and Wayne Xin Zhao. 2024 · 2024

Closest in time.

Exact and Efficient Unlearning for Large Language Model-based Recommendation

Original

Zhiyu Hu, Yang Zhang, Minghao Xiao, Wenjie Wang, Fuli Feng, and Xiangnan He. 2024 · 2024

Closest in time.