2020

Self-Supervised Graph Representation Learning via Global Context Prediction

Peng, Zhen, Dong, Yixiang, Luo, Minnan et al.

Understand

To take full advantage of fast-growing unlabeled networked data, this paper introduces a novel self-supervised strategy for graph representation learning by exploiting natural supervision provided by the data itself.

Inspired by human social behavior, we assume that the global context of each node is composed of all nodes in the graph since two arbitrary entities in a connected network could interact with each other via paths of varying length.
Based on this, we investigate whether the global context can be a source of free and effective supervisory signals for learning useful node representations.
Specifically, we randomly select pairs of nodes in a graph and train a well-designed neural net to predict the contextual position of one node relative to the other.

Built on

The class imbalance problem: A systematic study
Nathalie Japkowicz and Shaju Stephen · 2002
Earlier work this paper cites.
Visualizing data using t-sne
Laurens van der Maaten and Geoffrey Hinton · 2008
Earlier work this paper cites.
Understanding the difficulty of training deep feedforward neural networks
Xavier Glorot and Yoshua Bengio · 2010
Earlier work this paper cites.
Distributed representations of words and phrases and their compositionality
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean · 2013
Earlier work this paper cites.
Adam: A method for stochastic optimization
Original
Diederik P Kingma and Jimmy Ba · 2014
Earlier work this paper cites.
Deepwalk: Online learning of social representations
Bryan Perozzi, Rami Al-Rfou, and Steven Skiena · 2014
Earlier work this paper cites.
Unsupervised visual representation learning by context prediction
Carl Doersch, Abhinav Gupta, and Alexei A Efros · 2015
Earlier work this paper cites.
Unsupervised streaming feature selection in social media
Jundong Li, Xia Hu, Jiliang Tang, and Huan Liu · 2015
Earlier work this paper cites.
node2vec: Scalable feature learning for networks
Aditya Grover and Jure Leskovec · 2016
Earlier work this paper cites.
Semi-supervised classification with graph convolutional networks
Original
Thomas N Kipf and Max Welling · 2016
Earlier work this paper cites.
Learning graph representations with embedding propagation
Alberto Garcia Duran and Mathias Niepert · 2017
Earlier work this paper cites.

Similar

Inductive representation learning on large graphs
Will Hamilton, Zhitao Ying, and Jure Leskovec · 2017
Cited alongside, same era.
Graph attention networks
Original
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio · 2017
Cited alongside, same era.
Predicting multicellular function through multi-layer tissue networks
Marinka Zitnik and Jure Leskovec · 2017
Cited alongside, same era.
Fastgcn: fast learning with graph convolutional networks via importance sampling
Jie Chen, Tengfei Ma, and Cao Xiao · 2018
Cited alongside, same era.
Unsupervised representation learning by predicting image rotations
Spyros Gidaris, Praveer Singh, and Nikos Komodakis · 2018
Cited alongside, same era.
Regal: Representation learning-based graph alignment
Mark Heimann, Haoming Shen, Tara Safavi, and Danai Koutra · 2018
Cited alongside, same era.
Adaptive sampling towards fast graph representation learning
Wenbing Huang, Tong Zhang, Yu Rong, and Junzhou Huang · 2018
Cited alongside, same era.
Grasp2vec: Learning object representations from self-supervised grasping
Eric Jang, Coline Devin, Vincent Vanhoucke, and Sergey Levine · 2018
Cited alongside, same era.
Deeper insights into graph convolutional networks for semi-supervised learning
Qimai Li, Zhichao Han, and Xiao-Ming Wu · 2018
Cited alongside, same era.
Adaptive graph convolutional neural networks
Ruoyu Li, Sheng Wang, Feiyun Zhu, and Junzhou Huang · 2018
Cited alongside, same era.
Networks
Mark Newman · 2018
Cited alongside, same era.

Then

Network embedding as matrix factorization: Unifying deepwalk, line, pte, and node2vec
Jiezhong Qiu, Yuxiao Dong, Hao Ma, Jian Li, Kuansan Wang, and Jie Tang · 2018
Later among the works it cites.
Deep autoencoder-like nonnegative matrix factorization for community detection
Fanghua Ye, Chuan Chen, and Zibin Zheng · 2018
Later among the works it cites.
Gaan: Gated attention networks for learning on large and spatiotemporal graphs
Jiani Zhang, Xingjian Shi, Junyuan Xie, Hao Ma, Irwin King, and Dit-Yan Yeung · 2018
Later among the works it cites.
Graphite: Iterative generative modeling of graphs
Aditya Grover, Aaron Zweig, and Stefano Ermon · 2019
Later among the works it cites.
Self-supervised visual feature learning with deep neural networks: A survey
Original
Longlong Jing and Yingli Tian · 2019
Later among the works it cites.
Gmnn: Graph markov neural networks
Meng Qu, Yoshua Bengio, and Jian Tang · 2019
Later among the works it cites.
Multi-stage self-supervised learning for graph convolutional networks
Original
Ke Sun, Zhanxing Zhu, and Zhouchen Lin · 2019
Later among the works it cites.
Deep graph infomax
Petar Veličković, William Fedus, William L. Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm · 2019
Later among the works it cites.
Self-supervised dialogue learning
Original
Jiawei Wu, Xin Wang, and William Yang Wang · 2019
Later among the works it cites.
Graph wavelet neural network
Bingbing Xu, Huawei Shen, Qi Cao, Yunqi Qiu, and Xueqi Cheng · 2019
Later among the works it cites.
Attributed graph clustering via adaptive graph convolution
Xiaotong Zhang, Han Liu, Qimai Li, and Xiao-Ming Wu · 2019
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…