2021

SSFG: Stochastically Scaling Features and Gradients for Regularizing Graph Convolutional Networks

Zhang, Haimin, Xu, Min, Zhang, Guoqiang et al.

Understand

Graph convolutional networks have been successfully applied in various graph-based tasks.

In a typical graph convolutional layer, node features are updated by aggregating neighborhood information.
Repeatedly applying graph convolutions can cause the oversmoothing issue, i.e., node features at deep layers converge to similar values.
Previous studies have suggested that oversmoothing is one of the major issues that restrict the performance of graph convolutional networks.

Built on

S. J. Nowlan and G. E. Hinton, “Simplifying neural networks by soft weight-sharing,” Neural computation , vol. 4, no. 4, pp. 473–493, 1992
1992
Earlier work this paper cites.
S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural computation , vol. 9, no. 8, pp. 1735–1780, 1997
1997
Earlier work this paper cites.
V. Nair and G. E. Hinton, “Rectified linear units improve restricted boltzmann machines,” in ICML , 2010
2010
Earlier work this paper cites.
N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: a simple way to prevent neural networks from overfitting,” The journal of machine learning research , vol. 15, no. 1, pp. 1929–1958, 2014
2014
Earlier work this paper cites.
J. Bruna, W. Zaremba, A. Szlam, and Y. Lecun, “Spectral networks and locally connected networks on graphs,” in International Conference on Learning Representations (ICLR2014), CBLS, April 2014 , 2014
2014
Earlier work this paper cites.
D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980 , 2014
Original
2014
Earlier work this paper cites.
S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” arXiv preprint arXiv:1502.03167 , 2015
Original
2015
Earlier work this paper cites.
M. Defferrard, X. Bresson, and P. Vandergheynst, “Convolutional neural networks on graphs with fast localized spectral filtering,” in Advances in neural information processing systems , 2016, pp. 3844–3852
2016
Earlier work this paper cites.
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 770–778
2016
Earlier work this paper cites.
G. Klambauer, T. Unterthiner, A. Mayr, and S. Hochreiter, “Self-normalizing neural networks,” in Advances in neural information processing systems , 2017, pp. 971–980
2017
Earlier work this paper cites.
W. Hamilton, Z. Ying, and J. Leskovec, “Inductive representation learning on large graphs,” in Advances in neural information processing systems , 2017, pp. 1024–1034
2017
Earlier work this paper cites.
X. Bresson and T. Laurent, “Residual gated graph convnets,” arXiv preprint arXiv:1711.07553 , 2017
Original
2017
Earlier work this paper cites.
T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” in International Conference on Learning Representations (ICLR2017) , 2017
2017
Earlier work this paper cites.

Similar

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” arXiv preprint arXiv:1706.03762 , 2017
Original
2017
Cited alongside, same era.
D. Marcheggiani and I. Titov, “Encoding sentences with graph convolutional networks for semantic role labeling,” arXiv preprint arXiv:1703.04826 , 2017
Original
2017
Cited alongside, same era.
A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer, “Automatic differentiation in pytorch,” 2017
2017
Cited alongside, same era.
M. Zhang and Y. Chen, “Link prediction based on graph neural networks,” in Advances in Neural Information Processing Systems , 2018, pp. 5165–5175
2018
Cited alongside, same era.
Q. Li, Z. Han, and X.-M. Wu, “Deeper insights into graph convolutional networks for semi-supervised learning,” AAAI Conference on Artificial Intelligence , 2018
2018
Cited alongside, same era.
P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio, “Graph Attention Networks,” International Conference on Learning Representations , 2018, accepted as poster. [Online]. Available: https://openreview.net/forum?id=rJXMpikCZ
2018
Cited alongside, same era.
K. Xu, C. Li, Y. Tian, T. Sonobe, K.-i. Kawarabayashi, and S. Jegelka, “Representation learning on graphs with jumping knowledge networks,” in International Conference on Machine Learning . PMLR, 2018, pp. 5453–5462
2018
Cited alongside, same era.
J. Chen, T. Ma, and C. Xiao, “Fastgcn: fast learning with graph convolutional networks via importance sampling,” International Conference on Learning Representations , 2018
2018
Cited alongside, same era.
W. Huang, T. Zhang, Y. Rong, and J. Huang, “Adaptive sampling towards fast graph representation learning,” in Advances in neural information processing systems , 2018, pp. 4558–4567
2018
Cited alongside, same era.
J. Zhang, X. Shi, J. Xie, H. Ma, I. King, and D.-Y. Yeung, “Gaan: Gated attention networks for learning on large and spatiotemporal graphs,” UAI 2018 , 2018
2018
Cited alongside, same era.
S. Wu, Y. Tang, Y. Zhu, L. Wang, X. Xie, and T. Tan, “Session-based recommendation with graph neural networks,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 33, 2019, pp. 346–353
2019
Cited alongside, same era.
G. Li, M. Muller, A. Thabet, and B. Ghanem, “Deepgcns: Can gcns go as deep as cnns?” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2019, pp. 9267–9276
2019
Cited alongside, same era.

Then

L. Zhao and L. Akoglu, “Pairnorm: Tackling oversmoothing in gnns,” arXiv preprint arXiv:1909.12223 , 2019
Original
2019
Later among the works it cites.
M. Wang, L. Yu, D. Zheng, Q. Gan, Y. Gai, Z. Ye, M. Li, J. Zhou, Q. Huang, C. Ma et al. , “Deep graph library: Towards efficient and scalable deep learning on graphs,” arXiv preprint arXiv:1909.01315 , 2019
Original
2019
Later among the works it cites.
I. Chami, A. Wolf, D.-C. Juan, F. Sala, S. Ravi, and C. Ré, “Low-dimensional hyperbolic knowledge graph embeddings,” arXiv preprint arXiv:2005.00545 , 2020
Original
2020
Later among the works it cites.
V. P. Dwivedi, C. K. Joshi, T. Laurent, Y. Bengio, and X. Bresson, “Benchmarking graph neural networks,” arXiv preprint arXiv:2003.00982 , 2020
Original
2020
Later among the works it cites.
Y. Rong, W. Huang, T. Xu, and J. Huang, “Dropedge: Towards deep graph convolutional networks on node classification,” in International Conference on Learning Representations , 2020
2020
Later among the works it cites.
W. Feng, J. Zhang, Y. Dong, Y. Han, H. Luan, Q. Xu, Q. Yang, E. Kharlamov, and J. Tang, “Graph random neural networks for semi-supervised learning on graphs,” Advances in Neural Information Processing Systems , vol. 33, 2020
2020
Later among the works it cites.
C. Yang, R. Wang, S. Yao, S. Liu, and T. Abdelzaher, “Revisiting” over-smoothing” in deep gcns,” arXiv preprint arXiv:2003.13663 , 2020
Original
2020
Later among the works it cites.
Z. Wu, S. Pan, F. Chen, G. Long, C. Zhang, and S. Y. Philip, “A comprehensive survey on graph neural networks,” IEEE Transactions on Neural Networks and Learning Systems , 2020
2020
Later among the works it cites.
Y. Min, F. Wenkel, and G. Wolf, “Scattering gcn: Overcoming oversmoothness in graph convolutional networks,” arXiv preprint arXiv:2003.08414 , 2020
Original
2020
Later among the works it cites.
D. Chen, Y. Lin, W. Li, P. Li, J. Zhou, and X. Sun, “Measuring and relieving the over-smoothing problem for graph neural networks from the topological view,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 34, no. 04, 2020, pp. 3438–3445
2020
Later among the works it cites.
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly et al. , “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929 , 2020
Original
2020
Later among the works it cites.
W. Hu, M. Fey, M. Zitnik, Y. Dong, H. Ren, B. Liu, M. Catasta, and J. Leskovec, “Open graph benchmark: Datasets for machine learning on graphs,” arXiv preprint arXiv:2005.00687 , 2020
Original
2020
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…