2020

Understanding and Resolving Performance Degradation in Graph Convolutional Networks

Zhou, Kuangqi, Dong, Yanfei, Wang, Kaixin et al.

Understand

A Graph Convolutional Network (GCN) stacks several layers and in each layer performs a PROPagation operation (PROP) and a TRANsformation operation (TRAN) for learning node representations over graph-structured data.

Though powerful, GCNs tend to suffer performance drop when the model gets deep.
Previous works focus on PROPs to study and mitigate this issue, but the role of TRANs is barely investigated.
In this work, we study performance degradation of GCNs by experimentally examining how stacking only TRANs or PROPs works.

Built on

Collective classification in network data
Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad. 2008 · 2008
Earlier work this paper cites.
Rectified linear units improve restricted boltzmann machines. In ICML
Vinod Nair and Geoffrey E Hinton. 2010 · 2010
Earlier work this paper cites.
Batch normalization: Accelerating deep network training by reducing internal covariate shift
Original
Sergey Ioffe and Christian Szegedy. 2015 · 2015
Earlier work this paper cites.
Layer normalization
Original
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016 · 2016
Earlier work this paper cites.
Convolutional neural networks on graphs with fast localized spectral filtering. In Advances in neural information processing systems . 3844–3852
Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. 2016 · 2016
Earlier work this paper cites.
Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition . 770–778
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016 · 2016
Earlier work this paper cites.
Semi-supervised classification with graph convolutional networks
Original
Thomas N Kipf and Max Welling. 2016 · 2016
Earlier work this paper cites.
Benefits of depth in neural networks
Original
Matus Telgarsky. 2016 · 2016
Earlier work this paper cites.
Neural message passing for quantum chemistry. In Proceedings of the 34th International Conference on Machine Learning-Volume 70 . JMLR. org, 1263–1272
Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. 2017 · 2017
Earlier work this paper cites.
Inductive representation learning on large graphs. In Advances in neural information processing systems . 1024–1034
Will Hamilton, Zhitao Ying, and Jure Leskovec. 2017 · 2017
Earlier work this paper cites.
Self-normalizing neural networks. In Advances in neural information processing systems . 971–980
Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. 2017 · 2017
Earlier work this paper cites.
Attention is all you need. In Advances in neural information processing systems . 5998–6008
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017 · 2017
Earlier work this paper cites.

Similar

Graph attention networks
Original
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2017 · 2017
Cited alongside, same era.
Predict then propagate: Graph neural networks meet personalized pagerank
Original
Johannes Klicpera, Aleksandar Bojchevski, and Stephan Günnemann. 2018 · 2018
Cited alongside, same era.
Deeper insights into graph convolutional networks for semi-supervised learning. In Thirty-Second AAAI Conference on Artificial Intelligence
Qimai Li, Zhichao Han, and Xiao-Ming Wu. 2018 · 2018
Cited alongside, same era.
Pitfalls of graph neural network evaluation
Original
Oleksandr Shchur, Maximilian Mumme, Aleksandar Bojchevski, and Stephan Günnemann. 2018 · 2018
Cited alongside, same era.
Representation learning on graphs with jumping knowledge networks
Original
Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka. 2018 · 2018
Cited alongside, same era.
Graph neural networks: A review of methods and applications
Original
Jie Zhou, Ganqu Cui, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. 2018 · 2018
Cited alongside, same era.
Measuring and Relieving the Over-smoothing Problem for Graph Neural Networks from the Topological View
Original
Deli Chen, Yankai Lin, Wei Li, Peng Li, Jie Zhou, and Xu Sun. 2019 · 2019
Cited alongside, same era.
Fast Graph Representation Learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds
Matthias Fey and Jan E. Lenssen. 2019 · 2019
Cited alongside, same era.
Deepgcns: Can gcns go as deep as cnns?. In Proceedings of the IEEE International Conference on Computer Vision . 9267–9276
Guohao Li, Matthias Muller, Ali Thabet, and Bernard Ghanem. 2019 · 2019
Cited alongside, same era.
On asymptotic behaviors of graph cnns from dynamical systems perspective
Original
Kenta Oono and Taiji Suzuki. 2019 · 2019
Cited alongside, same era.
Multi-stage self-supervised learning for graph convolutional networks
Original
Ke Sun, Zhanxing Zhu, and Zhouchen Lin. 2019 · 2019
Cited alongside, same era.

Then

GraphMix: Regularized Training of Graph Neural Networks for Semi-Supervised Learning
Original
Vikas Verma, Meng Qu, Alex Lamb, Yoshua Bengio, Juho Kannala, and Jian Tang. 2019 · 2019
Later among the works it cites.
Simplifying graph convolutional networks
Original
Felix Wu, Tianyi Zhang, Amauri Holanda de Souza Jr, Christopher Fifty, Tao Yu, and Kilian Q Weinberger. 2019 · 2019
Later among the works it cites.
PairNorm: Tackling Oversmoothing in GNNs
Original
Lingxiao Zhao and Leman Akoglu. 2019 · 2019
Later among the works it cites.
Simple and Deep Graph Convolutional Networks
Original
Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. 2020 · 2020
Closest in time.
Open graph benchmark: Datasets for machine learning on graphs
Original
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. 2020 · 2020
Closest in time.
Combining Label Propagation and Simple Models Out-performs Graph Neural Networks
Original
Qian Huang, Horace He, Abhay Singh, Ser-Nam Lim, and Austin R Benson. 2020 · 2020
Closest in time.
Residual Correlation in Graph Neural Network Regression. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining . 588–598
Junteng Jia and Austion R Benson. 2020 · 2020
Closest in time.
Deepergcn: All you need to train deeper gcns
Original
Guohao Li, Chenxin Xiong, Ali Thabet, and Bernard Ghanem. 2020 · 2020
Closest in time.
Wiki-CS: A Wikipedia-Based Benchmark for Graph Neural Networks
Original
Péter Mernyei and Cătălina Cangea. 2020 · 2020
Closest in time.
Geom-GCN: Geometric Graph Convolutional Networks. In International Conference on Learning Representations
Hongbin Pei, Bingzhe Wei, Kevin Chen-Chuan Chang, Yu Lei, and Bo Yang. 2020 · 2020
Closest in time.
DropEdge: Towards Deep Graph Convolutional Networks on Node Classification. In International Conference on Learning Representations
Yu Rong, Wenbing Huang, Tingyang Xu, and Junzhou Huang. 2020 · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…