2021

Graph Barlow Twins: A self-supervised representation learning framework for graphs

Bielak, Piotr, Kajdanowicz, Tomasz, Chawla, Nitesh V.

Understand

The self-supervised learning (SSL) paradigm is an essential exploration area, which tries to eliminate the need for expensive data labeling.

Despite the great success of SSL methods in computer vision and natural language processing, most of them employ contrastive learning objectives that require negative samples, which are hard to define.
This becomes even more challenging in the case of graphs and is a bottleneck for achieving robust representations.
To overcome such limitations, we propose a framework for self-supervised graph representation learning - Graph Barlow Twins, which utilizes a cross-correlation-based loss function instead of negative samples.

Built on

Schmidhuber, J.: Making the world differentiable: On using fully recurrent self-supervised neural networks for dynamic reinforcement learning and planning in non-stationary environments. Tech. Rep. FKI-126-90 (revised), Institut für Informatik, Technische Universität München (November 1990)
1990
Earlier work this paper cites.
Schmidhuber, J., Eldracher, M., Foltin, B.: Semilinear predictability minimization produces well-known feature detectors. Neural Computation 8
1996
Earlier work this paper cites.
Deco, G., Parra, L.: Non-linear feature extraction by redundancy reduction in an unsupervised stochastic neural network. Neural Netw. 10
1997
Earlier work this paper cites.
Zhu, Y., Xu, Y., Yu, F., Liu, Q., Wu, S., Wang, L.: Deep Graph Contrastive Representation Learning. In: ICML Workshop on Graph Representation Learning and Beyond (2020), http://arxiv.org/abs/2006.04131
Original
2006
Earlier work this paper cites.
Zhu, Y., Xu, Y., Yu, F., Liu, Q., Wu, S., Wang, L.: Graph contrastive learning with adaptive augmentation. CoRR abs/2010.14945
Original
2010
Earlier work this paper cites.
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., Duchesnay, E.: Scikit-learn: Machine learning in Python. Journal of Machine Learning Research 12
2011
Earlier work this paper cites.
Perozzi, B., Al-Rfou, R., Skiena, S.: Deepwalk: Online learning of social representations. In: Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. pp. 701–710. KDD ’14, ACM, New York, NY, USA (2014). https://doi.org/10.1145/2623330.2623732, http://doi.acm.org/10.1145/2623330.2623732
2014
Earlier work this paper cites.
McAuley, J., Targett, C., Shi, Q., van den Hengel, A.: Image-based recommendations on styles and substitutes. In: Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval. p. 43–52. SIGIR ’15, Association for Computing Machinery, New York, NY, USA (2015). https://doi.org/10.1145/2766462.2767755, https://doi.org/10.1145/2766462.2767755
2015
Earlier work this paper cites.
Sinha, A., Shen, Z., Song, Y., Ma, H., Eide, D., Hsu, B.J.P., Wang, K.: An overview of microsoft academic service (mas) and applications. In: Proceedings of the 24th International Conference on World Wide Web. p. 243–246. WWW ’15 Companion, Association for Computing Machinery, New York, NY, USA (2015). https://doi.org/10.1145/2740908.2742839, https://doi.org/10.1145/2740908.2742839
2015
Earlier work this paper cites.
Tang, J., Qu, M., Wang, M., Zhang, M., Yan, J., Mei, Q.: Line: Large-scale information network embedding. In: WWW. ACM (2015)
2015
Earlier work this paper cites.
Grover, A., Leskovec, J.: node2vec: Scalable feature learning for networks. In: KDD (2016)
2016
Earlier work this paper cites.

Similar

Kipf, T.N., Welling, M.: Variational graph auto-encoders. NIPS Workshop on Bayesian Deep Learning (2016)
2016
Cited alongside, same era.
Ballé, J., Laparra, V., Simoncelli, E.P.: End-to-end optimized image compression. In: Int’l Conf on Learning Representations (ICLR). Toulon, France (April 2017), https://arxiv.org/abs/1611.01704 , available at http://arxiv.org/abs/1611.01704
Original
2017
Cited alongside, same era.
Hamilton, W.L., Ying, R., Leskovec, J.: Inductive representation learning on large graphs. In: NIPS (2017)
2017
Cited alongside, same era.
Kipf, T.N., Welling, M.: Semi-supervised classification with graph convolutional networks. In: International Conference on Learning Representations (ICLR) (2017)
2017
Cited alongside, same era.
Zitnik, M., Leskovec, J.: Predicting multicellular function through multi-layer tissue networks. Bioinformatics 33
2017
Cited alongside, same era.
Gugger, S., Howard, J.: Adamw and super-convergence is now the fastest way to train neural nets. (2018), https://www.fast.ai/2018/07/02/adam-weight-decay/
2018
Cited alongside, same era.
Veličković, P., Cucurull, G., Casanova, A., Romero, A., Liò, P., Bengio, Y.: Graph Attention Networks. International Conference on Learning Representations (2018), https://openreview.net/forum?id=rJXMpikCZ , accepted as poster
2018
Cited alongside, same era.
Fey, M., Lenssen, J.E.: Fast graph representation learning with PyTorch Geometric. In: ICLR Workshop on Representation Learning on Graphs and Manifolds (2019)
2019
Cited alongside, same era.
Qiao, S., Wang, H., Liu, C., Shen, W., Yuille, A.: Weight standardization. arXiv preprint arXiv:1903.10520 (2019)
Original
2019
Cited alongside, same era.
Veličković, P., Fedus, W., Hamilton, W.L., Liò, P., Bengio, Y., Hjelm, R.D.: Deep Graph Infomax. In: International Conference on Learning Representations (2019), https://openreview.net/forum?id=rklz9iAcKQ
2019
Cited alongside, same era.

Then

Che, F., Yang, G., Zhang, D., Tao, J., Shao, P., Liu, T.: Self-supervised graph representation learning via bootstrapping. arXiv preprint arXiv:2011.05126 (2020)
Original
2020
Later among the works it cites.
Chen, X., He, K.: Exploring simple siamese representation learning (2020)
2020
Later among the works it cites.
Grill, J.B., Strub, F., Altché, F., Tallec, C., Richemond, P., Buchatskaya, E., Doersch, C., Avila Pires, B., Guo, Z., Gheshlaghi Azar, M., Piot, B., kavukcuoglu, k., Munos, R., Valko, M.: Bootstrap your own latent - a new approach to self-supervised learning. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M.F., Lin, H. (eds.) Advances in Neural Information Processing Systems. vol. 33, pp. 21271–21284. Curran Associates, Inc. (2020), https://proceedings.neurips.cc/paper/2020/file/f3ada80d5c4ee70142b17b8192b2958e-Paper.pdf
2020
Later among the works it cites.
Hu, W., Fey, M., Zitnik, M., Dong, Y., Ren, H., Liu, B., Catasta, M., Leskovec, J.: Open graph benchmark: Datasets for machine learning on graphs. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M.F., Lin, H. (eds.) Advances in Neural Information Processing Systems. vol. 33, pp. 22118–22133. Curran Associates, Inc. (2020), https://proceedings.neurips.cc/paper/2020/file/fb60d411a5c5b72b2e7d3527cfc84fd0-Paper.pdf
2020
Later among the works it cites.
Mernyei, P., Cangea, C.: Wiki-cs: A wikipedia-based benchmark for graph neural networks. arXiv preprint arXiv:2007.02901 (2020)
Original
2020
Later among the works it cites.
You, Y., Chen, T., Sui, Y., Chen, T., Wang, Z., Shen, Y.: Graph contrastive learning with augmentations. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M.F., Lin, H. (eds.) Advances in Neural Information Processing Systems. vol. 33, pp. 5812–5823. Curran Associates, Inc. (2020), https://proceedings.neurips.cc/paper/2020/file/3fe230348e9a12c13120749e3f9fa4cd-Paper.pdf
2020
Later among the works it cites.
Kuprieiev, R., Petrov, D., Pachhai, S., Redzyński, P., da Costa-Luis, C., Schepanovski, A., Rowlands, P., Shcheklein, I., Orpinel, J., Santos, F., Taskaya, B., Sharma, A., Zhanibek, Gao, Hodovic, D., Grigorev, A., Earl, nik123, Dash, N., Vyshnya, G., maykulkarni, Hora, M., Vera, Mangal, S., Baranowski, W., Wolff, C., Maslakov, A., Khamutov, A., Benoy, K.: Dvc: Data version control - git for data & models (May 2021). https://doi.org/10.5281/zenodo.4733984, https://doi.org/10.5281/zenodo.4733984
2021
Closest in time.
Thakoor, S., Tallec, C., Azar, M.G., Munos, R., Veličković, P., Valko, M.: Bootstrapped representation learning on graphs. In: ICLR 2021 Workshop on Geometrical and Topological Representation Learning (2021), https://openreview.net/forum?id=QrzVRAA49Ud
2021
Closest in time.
Tsai, Y.H.H., Bai, S., Morency, L.P., Salakhutdinov, R.: A note on connecting barlow twins with negative-sample-free contrastive learning (2021)
2021
Closest in time.
Zbontar, J., Jing, L., Misra, I., LeCun, Y., Deny, S.: Barlow twins: Self-supervised learning via redundancy reduction (2021)
2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…