2020

Graph Contrastive Learning with Augmentations

You, Yuning, Chen, Tianlong, Sui, Yongduo et al.

Understand

Generalizable, transferrable, and robust representation learning on graph-structured data remains a challenge for current graph neural networks (GNNs).

Unlike what has been developed for convolutional neural networks (CNNs) for image data, self-supervised learning and pre-training are less explored for GNNs.
In this paper, we propose a graph contrastive learning (GraphCL) framework for learning unsupervised representations of graph data.
We first design four types of graph augmentations to incorporate various priors.

Built on

Self-organizing neural network that discovers surfaces in random-dot stereograms
Suzanna Becker and Geoffrey E Hinton · 1992
Earlier work this paper cites.
To transfer or not to transfer
Michael T Rosenstein, Zvika Marx, Leslie Pack Kaelbling, and Thomas G Dietterich · 2005
Earlier work this paper cites.
The difficulty of training deep architectures and the effect of unsupervised pre-training
Dumitru Erhan, Pierre-Antoine Manzagol, Yoshua Bengio, Samy Bengio, and Pascal Vincent · 2009
Earlier work this paper cites.
Understanding the difficulty of training deep feedforward neural networks
Xavier Glorot and Yoshua Bengio · 2010
Earlier work this paper cites.
Semi-supervised classification with graph convolutional networks
Original
Thomas N Kipf and Max Welling · 2016
Earlier work this paper cites.
Variational graph auto-encoders
Original
Thomas N Kipf and Max Welling · 2016
Earlier work this paper cites.
Unsupervised learning of visual representations by solving jigsaw puzzles
Mehdi Noroozi and Paolo Favaro · 2016
Earlier work this paper cites.
Improved deep metric learning with multi-class n-pair loss objective
Kihyuk Sohn · 2016
Earlier work this paper cites.
node2vec: Scalable feature learning for networks
Aditya Grover and Jure Leskovec · 2016
Earlier work this paper cites.
Discriminative embeddings of latent variable models for structured data
Hanjun Dai, Bo Dai, and Le Song · 2016
Earlier work this paper cites.
Graph attention networks
Original
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio · 2017
Earlier work this paper cites.
Inductive representation learning on large graphs
Will Hamilton, Zhitao Ying, and Jure Leskovec · 2017
Earlier work this paper cites.
struc2vec: Learning node representations from structural identity
Leonardo FR Ribeiro, Pedro HP Saverese, and Daniel R Figueiredo · 2017
Earlier work this paper cites.
On sampling strategies for neural network-based collaborative filtering
Ting Chen, Yizhou Sun, Yue Shi, and Liangjie Hong · 2017
Earlier work this paper cites.
Neural message passing for quantum chemistry
Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl · 2017
Earlier work this paper cites.
graph2vec: Learning distributed representations of graphs
Original
Annamalai Narayanan, Mahinthan Chandramohan, Rajasekar Venkatesan, Lihui Chen, Yang Liu, and Shantanu Jaiswal · 2017
Earlier work this paper cites.
How powerful are graph neural networks?
Original
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka · 2018
Earlier work this paper cites.
Link prediction based on graph neural networks
Muhan Zhang and Yixin Chen · 2018
Earlier work this paper cites.
Hierarchical graph representation learning with differentiable pooling
Zhitao Ying, Jiaxuan You, Christopher Morris, Xiang Ren, Will Hamilton, and Jure Leskovec · 2018
Earlier work this paper cites.
Deeper insights into graph convolutional networks for semi-supervised learning
Qimai Li, Zhichao Han, and Xiao-Ming Wu · 2018
Earlier work this paper cites.
Prioritizing network communities
Marinka Zitnik, Jure Leskovec, et al · 2018
Earlier work this paper cites.
Deep graph infomax
Original
Petar Veličković, William Fedus, William L Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm · 2018
Earlier work this paper cites.
Unsupervised feature learning via non-parametric instance discrimination
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin · 2018
Earlier work this paper cites.

Similar

Semi-supervised learning on graphs with generative adversarial nets
Ming Ding, Jie Tang, and Jie Zhang · 2018
Cited alongside, same era.
Mine: mutual information neural estimation
Original
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and R Devon Hjelm · 2018
Cited alongside, same era.
Learning deep representations by mutual information estimation and maximization
Original
R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio · 2018
Cited alongside, same era.
Representation learning with contrastive predictive coding
Original
Aaron van den Oord, Yazhe Li, and Oriol Vinyals · 2018
Cited alongside, same era.
Adversarial attack on graph structured data
Original
Hanjun Dai, Hui Li, Tian Tian, Xin Huang, Lin Wang, Jun Zhu, and Le Song · 2018
Cited alongside, same era.
Adversarial attacks on neural networks for graph data
Daniel Zügner, Amir Akbarnejad, and Stephan Günnemann · 2018
Cited alongside, same era.
Sub2vec: Feature learning for subgraphs
Bijaya Adhikari, Yao Zhang, Naren Ramakrishnan, and B Aditya Prakash · 2018
Cited alongside, same era.
Layer-dependent importance sampling for training deep and large graph convolutional networks
Difan Zou, Ziniu Hu, Yewen Wang, Song Jiang, Yizhou Sun, and Quanquan Gu · 2019
Cited alongside, same era.
Pre-training graph neural networks
Original
Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec · 2019
Cited alongside, same era.
Graph neural networks exponentially lose expressive power for node classification
Kenta Oono and Taiji Suzuki · 2019
Cited alongside, same era.
Scaling and benchmarking self-supervised visual representation learning
Original
Priya Goyal, Dhruv Mahajan, Abhinav Gupta, and Ishan Misra · 2019
Cited alongside, same era.
Revisiting self-supervised visual representation learning
Original
Alexander Kolesnikov, Xiaohua Zhai, and Lucas Beyer · 2019
Cited alongside, same era.

Then

Mixmatch: A holistic approach to semi-supervised learning
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel · 2019
Later among the works it cites.
Deep graph infomax
Petar Velickovic, William Fedus, William L Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm · 2019
Later among the works it cites.
Heterogeneous deep graph infomax
Original
Yuxiang Ren, Bo Liu, Chao Huang, Peng Dai, Liefeng Bo, and Jiawei Zhang · 2019
Later among the works it cites.
Are powerful graph neural nets necessary? a dissection on graph classification
Original
Ting Chen, Song Bian, and Yizhou Sun · 2019
Later among the works it cites.
Towards deeper graph neural networks
Meng Liu, Hongyang Gao, and Shuiwang Ji · 2020
Closest in time.
Benchmarking graph neural networks
Original
Vijay Prakash Dwivedi, Chaitanya K Joshi, Thomas Laurent, Yoshua Bengio, and Xavier Bresson · 2020
Closest in time.
A simple framework for contrastive learning of visual representations
Original
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton · 2020
Closest in time.
Open graph benchmark: Datasets for machine learning on graphs
Original
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec · 2020
Closest in time.
Momentum contrast for unsupervised visual representation learning
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick · 2020
Closest in time.
Adversarial robustness: From self-supervised pre-training to fine-tuning
Original
Tianlong Chen, Sijia Liu, Shiyu Chang, Yu Cheng, Lisa Amini, and Zhangyang Wang · 2020
Closest in time.
When does self-supervision help graph convolutional networks?
Original
Yuning You, Tianlong Chen, Zhangyang Wang, and Yang Shen · 2020
Closest in time.
Self-supervised learning on graphs: Deep insights and new direction
Original
Wei Jin, Tyler Derr, Haochen Liu, Yiqi Wang, Suhang Wang, Zitao Liu, and Jiliang Tang · 2020
Closest in time.
Self-supervised training of graph convolutional networks
Original
Qikui Zhu, Bo Du, and Pingkun Yan · 2020
Closest in time.
Graph-bert: Only attention is needed for learning graph representations
Original
Jiawei Zhang, Haopeng Zhang, Li Sun, and Congying Xia · 2020
Closest in time.
Gpt-gnn: Generative pre-training of graph neural networks
Ziniu Hu, Yuxiao Dong, Kuansan Wang, Kai-Wei Chang, and Yizhou Sun · 2020
Closest in time.
Self-supervised learning: Generative or contrastive
Original
Xiao Liu, Fanjin Zhang, Zhenyu Hou, Zhaoyu Wang, Li Mian, Jing Zhang, and Jie Tang · 2020
Closest in time.
Self-supervised graph representation learning via global context prediction
Original
Zhen Peng, Yixiang Dong, Minnan Luo, Xiao-Ming Wu, and Qinghua Zheng · 2020
Closest in time.
Unsupervised attributed multiplex network embedding
Chanyoung Park, Donghyun Kim, Jiawei Han, and Hwanjo Yu · 2020
Closest in time.
Graph representation learning via graphical mutual information maximization
Zhen Peng, Wenbing Huang, Minnan Luo, Qinghua Zheng, Yu Rong, Tingyang Xu, and Junzhou Huang · 2020
Closest in time.
Contrastive multi-view representation learning on graphs
Original
Kaveh Hassani and Amir Hosein Khasahmadi · 2020
Closest in time.
Gcc: Graph contrastive coding for graph neural network pre-training
Jiezhong Qiu, Qibin Chen, Yuxiao Dong, Jing Zhang, Hongxia Yang, Ming Ding, Kuansan Wang, and Jie Tang · 2020
Closest in time.
Tudataset: A collection of benchmark datasets for learning with graphs
Christopher Morris, Nils M. Kriege, Franka Bause, Kristian Kersting, Petra Mutzel, and Marion Neumann · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…