2022

Transformer for Graphs: An Overview from Architecture Perspective

Min, Erxue, Chen, Runfa, Bian, Yatao et al.

Understand

Recently, Transformer model, which has achieved great success in many artificial intelligence fields, has demonstrated its great potential in modeling graph-structured data.

Till now, a great variety of Transformers has been proposed to adapt to the graph-structured data.
However, a comprehensive literature review and systematical evaluation of these Transformer variants for graphs are still unavailable.
It's imperative to sort out the existing Transformer models for graphs and systematically investigate their effectiveness on various graph tasks.

Built on

Graph Clustering
Charu C. Aggarwal · 2010
Earlier work this paper cites.
Computer vision: A modern approach
David Forsyth and Jean Ponce · 2011
Earlier work this paper cites.
Layer normalization
Original
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton · 2016
Earlier work this paper cites.
Semi-supervised classification with graph convolutional networks
Original
Thomas N Kipf and Max Welling · 2016
Earlier work this paper cites.
Revisiting semi-supervised learning with graph embeddings
Zhilin Yang, William Cohen, and Ruslan Salakhudinov · 2016
Earlier work this paper cites.
Neural message passing for quantum chemistry
Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl · 2017
Earlier work this paper cites.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin · 2017
Earlier work this paper cites.
Graph attention networks
Original
Petar Velivckovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio · 2017
Earlier work this paper cites.
How powerful are graph neural networks?
Original
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka · 2018
Earlier work this paper cites.
Representation learning on graphs with jumping knowledge networks
Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka · 2018
Earlier work this paper cites.
Link prediction based on graph neural networks
Muhan Zhang and Yixin Chen · 2018
Earlier work this paper cites.
Provably powerful graph networks
Haggai Maron, Heli Ben-Hamu, Hadar Serviansky, and Yaron Lipman · 2019
Earlier work this paper cites.
Universal graph transformer self-attention networks
Original
Dai Quoc Nguyen, Tu Dinh Nguyen, and Dinh Phung · 2019
Earlier work this paper cites.
Deep learning for audio signal processing
Hendrik Purwins, Bo Li, Tuomas Virtanen, Jan Schlter, Shuo-Yiin Chang, and Tara Sainath · 2019
Earlier work this paper cites.
Dropedge: Towards deep graph convolutional networks on node classification
Original
Yu Rong, Wenbing Huang, Tingyang Xu, and Junzhou Huang · 2019
Earlier work this paper cites.

Similar

Novel positional encodings to enable tree-based transformers
Vighnesh Shiv and Chris Quirk · 2019
Cited alongside, same era.
Transformer dissection: A unified understanding of transformer’s attention via the lens of kernel
Original
Yao-Hung Hubert Tsai, Shaojie Bai, Makoto Yamada, Louis-Philippe Morency, and Ruslan Salakhutdinov · 2019
Cited alongside, same era.
Self-attention with structural position representations
Original
Xing Wang, Zhaopeng Tu, Longyue Wang, and Shuming Shi · 2019
Cited alongside, same era.
Gresnet: Graph residual network for reviving deep gnns from suspended animation
Original
Jiawei Zhang and Lin Meng · 2019
Cited alongside, same era.
Pairnorm: Tackling oversmoothing in gnns
Original
Lingxiao Zhao and Leman Akoglu · 2019
Cited alongside, same era.
Modeling graph structure in transformer for better amr-to-text generation
Original
Jie Zhu, Junhui Li, Muhua Zhu, Longhua Qian, Min Zhang, and Guodong Zhou · 2019
Cited alongside, same era.
Graph transformer for graph-to-sequence learning
Deng Cai and Wai Lam · 2020
Cited alongside, same era.
Convolutional kernel networks for graph-structured data
Dexiong Chen, Laurent Jacob, and Julien Mairal · 2020
Cited alongside, same era.
A generalization of transformer networks to graphs
Original
Vijay Prakash Dwivedi and Xavier Bresson · 2020
Cited alongside, same era.
Se (3)-transformers: 3d roto-translation equivariant attention networks
Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling · 2020
Cited alongside, same era.
Time series analysis
James Douglas Hamilton · 2020
Cited alongside, same era.
Interpretable rumor detection in microblogs by attending to user interactions
Ling Min Serena Khoo, Hai Leong Chieu, Zhong Qian, and Jing Jiang · 2020
Cited alongside, same era.

Then

Heterogeneous graph transformer for graph-to-sequence learning
Shaowei Yao, Tianming Wang, and Xiaojun Wan · 2020
Later among the works it cites.
Deep graph neural networks with shallow subgraph samplers
Original
Hanqing Zeng, Muhan Zhang, Yinglong Xia, Ajitesh Srivastava, Andrey Malevich, Rajgopal Kannan, Viktor Prasanna, Long Jin, and Ren Chen · 2020
Later among the works it cites.
Graph-bert: Only attention is needed for learning graph representations
Original
Jiawei Zhang, Haopeng Zhang, Congying Xia, and Li Sun · 2020
Later among the works it cites.
Open catalyst 2020 (oc20) dataset and community challenges
Lowik Chanussot*, Abhishek Das*, Siddharth Goyal*, Thibaut Lavril*, Muhammed Shuaibi*, Morgane Riviere, Kevin Tran, Javier Heras-Domingo, Caleb Ho, Weihua Hu, Aini Palizhati, Anuroop Sriram, Brandon Wood, Junwoong Yoon, Devi Parikh, C. Lawrence Zitnick, and Zachary Ulissi · 2021
Later among the works it cites.
Ogb-lsc: A large-scale challenge for machine learning on graphs
Original
Weihua Hu, Matthias Fey, Hongyu Ren, Maho Nakata, Yuxiao Dong, and Jure Leskovec · 2021
Later among the works it cites.
Edge-augmented graph transformers: Global self-attention is enough for graphs
Original
Md Shamim Hussain, Mohammed J Zaki, and Dharmashankar Subramanian · 2021
Later among the works it cites.
Representing long-range context for graph neural networks with global attention
Paras Jain, Zhanghao Wu, Matthew Wright, Azalia Mirhoseini, Joseph E Gonzalez, and Ion Stoica · 2021
Later among the works it cites.
Rethinking graph transformers with spectral attention
Original
Devin Kreuzer, Dominique Beaini, William L Hamilton, Vincent Létourneau, and Prudencio Tossou · 2021
Later among the works it cites.
Mesh graphormer
Original
Kevin Lin, Lijuan Wang, and Zicheng Liu · 2021
Later among the works it cites.
Graphit: Encoding graph structure in transformers
Original
Grégoire Mialon, Dexiong Chen, Margot Selosse, and Julien Mairal · 2021
Later among the works it cites.
Do transformers really perform bad for graph representation?
Original
Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie-Yan Liu · 2021
Later among the works it cites.
Gophormer: Ego-graph transformer for node classification
Original
Jianan Zhao, Chaozhuo Li, Qianlong Wen, Yiqi Wang, Yuming Liu, Hao Sun, Xing Xie, and Yanfang Ye · 2021
Later among the works it cites.
Masked transformer for neighhourhood-aware click-through rate prediction
Original
Erxue Min, Yu Rong, Tingyang Xu, Yatao Bian, Peilin Zhao, Junzhou Huang, Da Luo, Kangyi Lin, and Sophia Ananiadou · 2022
Closest in time.
Equivariant transformers for neural network based molecular potentials
Philipp Thölke and Gianni de Fabritiis · 2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…