2023

Knowledge Distillation on Graphs: A Survey

Tian, Yijun, Pei, Shichao, Zhang, Xiangliang et al.

Understand

Graph Neural Networks (GNNs) have attracted tremendous attention by demonstrating their capability to handle graph data.

However, they are difficult to be deployed in resource-limited devices due to model sizes and scalability constraints imposed by the multi-hop data dependency.
In addition, real-world graphs usually possess complex structural information and features.
Therefore, to improve the applicability of GNNs and fully encode the complicated topological information, knowledge distillation on graphs (KDG) has been introduced to build a smaller yet effective model and exploit more knowledge from data, leading to model compression and performance improvement.

Built on

Distilling the knowledge in a neural network
Original
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al · 2015
Earlier work this paper cites.
Semi-supervised classification with graph convolutional networks
Thomas N. Kipf and Max Welling · 2017
Earlier work this paper cites.
Knowledge graph embedding: A survey of approaches and applications
Quan Wang, Zhendong Mao, Bin Wang, and Li Guo · 2017
Earlier work this paper cites.
Graph attention networks
Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio · 2018
Earlier work this paper cites.
Variational information distillation for knowledge transfer
Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D Lawrence, and Zhenwen Dai · 2019
Earlier work this paper cites.
Adaptive regularization of labels
Original
Qianggang Ding, Sifan Wu, Hao Sun, Jiadong Guo, and Shu-Tao Xia · 2019
Earlier work this paper cites.
Graph neural networks for social recommendation
Wenqi Fan, Yao Ma, Qing Li, Yuan He, Eric Zhao, Jiliang Tang, and Dawei Yin · 2019
Earlier work this paper cites.
Knowledge transfer via distillation of activation boundaries formed by hidden neurons
Byeongho Heo, Minsik Lee, Sangdoo Yun, and Jin Young Choi · 2019
Earlier work this paper cites.
Deepgcns: Can gcns go as deep as cnns?
Guohao Li, Matthias Muller, Ali Thabet, and Bernard Ghanem · 2019
Earlier work this paper cites.
Structured knowledge distillation for semantic segmentation
Yifan Liu, Ke Chen, Chris Liu, Zengchang Qin, Zhenbo Luo, and Jingdong Wang · 2019
Earlier work this paper cites.
When does label smoothing help?
Rafael Muller, Simon Kornblith, and Geoffrey E Hinton · 2019
Earlier work this paper cites.
Towards understanding knowledge distillation
Mary Phuong and Christoph Lampert · 2019
Earlier work this paper cites.
Patient knowledge distillation for bert model compression
Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu · 2019
Earlier work this paper cites.
How powerful are graph neural networks?
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka · 2019
Earlier work this paper cites.
Be your own teacher: Improve the performance of convolutional neural networks via self distillation
Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma · 2019
Earlier work this paper cites.
Towards understanding ensemble, knowledge distillation and self-distillation in deep learning
Zeyuan Allen-Zhu and Yuanzhi Li · 2020
Earlier work this paper cites.
Rumor detection on social media with bi-directional graph convolutional networks
Tian Bian, Xi Xiao, Tingyang Xu, Peilin Zhao, Wenbing Huang, Yu Rong, and Junzhou Huang · 2020
Earlier work this paper cites.
Open graph benchmark: Datasets for machine learning on graphs
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec · 2020
Earlier work this paper cites.
Graph structure learning for robust graph neural networks
Wei Jin, Yao Ma, Xiaorui Liu, Xianfeng Tang, Suhang Wang, and Jiliang Tang · 2020
Earlier work this paper cites.
Continual learning for robotics: Definition, framework, learning strategies, opportunities and challenges
Timothee Lesort, Vincenzo Lomonaco, Andrei Stoian, Davide Maltoni, David Filliat, and Natalia Diaz-Rodriguez · 2020
Earlier work this paper cites.
Graphon neural networks and the transferability of graph neural networks
Luana Ruiz, Luiz Chamon, and Alejandro Ribeiro · 2020
Earlier work this paper cites.
A comprehensive survey on graph neural networks
Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip · 2020
Earlier work this paper cites.
Tinygnn: Learning efficient graph neural networks
Bencheng Yan, Chaokun Wang, Gaoyang Guo, and Yunkai Lou · 2020
Earlier work this paper cites.

Similar

Distilling knowledge from graph convolutional networks
Yiding Yang, Jiayan Qiu, Mingli Song, Dacheng Tao, and Xinchao Wang · 2020
Cited alongside, same era.
Reliable data distillation on graph convolutional network
Wentao Zhang, Xupeng Miao, Yingxia Shao, Jiawei Jiang, Lei Chen, Olivier Ruas, and Bin Cui · 2020
Cited alongside, same era.
A unified lottery ticket hypothesis for graph neural networks
Tianlong Chen, Yongduo Sui, Xuxi Chen, Aston Zhang, and Zhangyang Wang · 2021
Cited alongside, same era.
On self-distilling graph neural network
Yuzhao Chen, Yatao Bian, Xi Xiao, Yu Rong, Tingyang Xu, and Junzhou Huang · 2021
Cited alongside, same era.
Graph-free knowledge distillation for graph neural networks
Xiang Deng and Zhongfei Zhang · 2021
Cited alongside, same era.
Knowledge distillation: A survey
Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao · 2021
Cited alongside, same era.
Towards black-box explainability with gaussian discriminant knowledge distillation
Anselm Haselhoff, Jan Kronenberger, Fabian Kuppers, and Jonas Schneider · 2021
Cited alongside, same era.
Knowledge graphs
Aidan Hogan, Eva Blomqvist, Michael Cochez, Claudia d’Amato, Gerard de Melo, Claudio Gutierrez, Sabrina Kirrane, Jose Emilio Labra Gayo, Roberto Navigli, Sebastian Neumaier, et al · 2021
Cited alongside, same era.
Graph condensation for graph neural networks
Wei Jin, Lingxiao Zhao, Shichang Zhang, Yozen Liu, Jiliang Tang, and Neil Shah · 2021
Cited alongside, same era.
Distilling meta knowledge on heterogeneous graph for illicit drug trafficker detection on social media
Yiyue Qian, Yiming Zhang, Yanfang Ye, Chuxu Zhang, et al · 2021
Cited alongside, same era.
Transferring knowledge distillation for multilingual social event detection
Original
Jiaqian Ren, Hao Peng, Lei Jiang, Jia Wu, Yongxin Tong, Lihong Wang, Xu Bai, Bo Wang, and Qiang Yang · 2021
Cited alongside, same era.
Edge: Enriching knowledge graph embeddings with external text
Saed Rezayi, Handong Zhao, Sungchul Kim, Ryan A Rossi, Nedim Lipka, and Sheng Li · 2021
Cited alongside, same era.

Then

Knowledge distillation as efficient pre-training: Faster convergence, higher data-efficiency, and better transferability
Ruifei He, Shuyang Sun, Jihan Yang, Song Bai, and Xiaojuan Qi · 2022
Later among the works it cites.
On representation knowledge distillation for graph neural networks
Chaitanya K Joshi, Fayao Liu, Xu Xun, Jie Lin, and Chuan Sheng Foo · 2022
Later among the works it cites.
Distilling knowledge on text graph for social media attribute inference
Quan Li, Xiaoting Li, Lingwei Chen, and Dinghao Wu · 2022
Later among the works it cites.
Egnn: Constructing explainable graph neural networks via knowledge distillation
Yuan Li, Li Liu, Guoyin Wang, Yong Du, and Penggang Chen · 2022
Later among the works it cites.
Hierarchical spatio-temporal graph neural networks for pandemic forecasting
Yihong Ma, Patrick Gerard, Yijun Tian, Zhichun Guo, and Nitesh V Chawla · 2022
Later among the works it cites.
3d infomax improves gnns for molecular property prediction
Hannes Stark, Dominique Beaini, Gabriele Corso, Prudencio Tossou, Christian Dallago, Stephan Gunnemann, and Pietro Lio · 2022
Later among the works it cites.
Reciperec: A heterogeneous graph learning model for recipe recommendation
Yijun Tian, Chuxu Zhang, Zhichun Guo, Chao Huang, Ronald Metoyer, and Nitesh V Chawla · 2022
Later among the works it cites.
Recipe2vec: Multi-modal recipe representation learning with graph neural networks
Yijun Tian, Chuxu Zhang, Zhichun Guo, Yihong Ma, Ronald Metoyer, and Nitesh V Chawla · 2022
Later among the works it cites.
Recipe recommendation with hierarchical graph attention network
Yijun Tian, Chuxu Zhang, Ronald Metoyer, and Nitesh V Chawla · 2022
Later among the works it cites.
Collaborative knowledge distillation for heterogeneous information network embedding
Can Wang, Sheng Zhou, Kang Yu, Defang Chen, Bolang Li, Yan Feng, and Chun Chen · 2022
Later among the works it cites.
Knowledge distillation improves graph structure augmentation for graph neural networks
Lirong Wu, Haitao Lin, Yufei Huang, and Stan Z Li · 2022
Later among the works it cites.
Geometric knowledge distillation: Topology compression for graph neural networks
Chenxiao Yang, Qitian Wu, and Junchi Yan · 2022
Later among the works it cites.
Sail: Self-augmented graph contrastive learning
Lu Yu, Shichao Pei, Lizhong Ding, Jun Zhou, Longfei Li, Chuxu Zhang, and Xiangliang Zhang · 2022
Later among the works it cites.
Lte4g: Long-tail experts for graph neural networks
Sukwon Yun, Kibum Kim, Kanghoon Yoon, and Chanyoung Park · 2022
Later among the works it cites.
Multi-scale distillation from multiple graph neural networks
Chunhai Zhang, Jie Liu, Kai Dang, and Wenzheng Zhang · 2022
Later among the works it cites.
Graph-less neural networks: Teaching old mlps new tricks via distillation
Shichang Zhang, Yozen Liu, Yizhou Sun, and Neil Shah · 2022
Later among the works it cites.
Cold brew: Distilling graph node representations with incomplete or missing neighborhoods
Wenqing Zheng, Edward W Huang, Nikhil Rao, Sumeet Katariya, Zhangyang Wang, and Karthik Subbian · 2022
Later among the works it cites.
Data-free adversarial knowledge distillation for graph neural networks
Yuanxin Zhuang, Lingjuan Lyu, Chuan Shi, Carl Yang, and Lichao Sun · 2022
Later among the works it cites.
Reliant: Fair knowledge distillation for graph neural networks
Yushun Dong, Binchi Zhang, Yiling Yuan, Na Zou, Qi Wang, and Jundong Li · 2023
Closest in time.
Boosting graph neural networks via adaptive knowledge distillation
Zhichun Guo, Chunhui Zhang, Yujie Fan, Yijun Tian, Chuxu Zhang, and Nitesh Chawla · 2023
Closest in time.
T2-gnn: Graph neural networks for graphs with incomplete features and structure via teacher-student distillation
Cuiying Huo, Di Jin, Yawen Li, Dongxiao He, Yu-Bin Yang, and Lingfei Wu · 2023
Closest in time.
Heterogeneous graph masked autoencoders
Yijun Tian, Kaiwen Dong, Chunhui Zhang, Chuxu Zhang, and Nitesh V Chawla · 2023
Closest in time.
Nosmog: Learning noise-robust and structure-aware mlps on graphs
Yijun Tian, Chuxu Zhang, Zhichun Guo, Xiangliang Zhang, and Nitesh V Chawla · 2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…