2023

Structure Pretraining and Prompt Tuning for Knowledge Graph Transfer

Zhang, Wen, Zhu, Yushan, Chen, Mingyang et al.

Understand

Knowledge graphs (KG) are essential background knowledge providers in many tasks.

When designing models for KG-related tasks, one of the key tasks is to devise the Knowledge Representation and Fusion (KRF) module that learns the representation of elements from KGs and fuses them with task representations.
While due to the difference of KGs and perspectives to be considered during fusion across tasks, duplicate and ad hoc KRF modules design are conducted among tasks.
In this paper, we propose a novel knowledge graph pretraining model KGTransformer that could serve as a uniform KRF module in diverse KG-related tasks.

Built on

Toward an Architecture for Never-Ending Language Learning. In AAAI . AAAI Press
Andrew Carlson, Justin Betteridge, Bryan Kisiel, Burr Settles, Estevam R. Hruschka Jr., and Tom M. Mitchell. 2010 · 2010
Earlier work this paper cites.
Translating Embeddings for Modeling Multi-relational Data. In NIPS . 2787–2795
Antoine Bordes, Nicolas Usunier, Alberto García-Durán, Jason Weston, and Oksana Yakhnenko. 2013 · 2013
Earlier work this paper cites.
DeViSE: A Deep Visual-Semantic Embedding Model. In NIPS . 2121–2129
Andrea Frome, Gregory S. Corrado, Jonathon Shlens, Samy Bengio, Jeffrey Dean, Marc’Aurelio Ranzato, and Tomás Mikolov. 2013 · 2013
Earlier work this paper cites.
Adam: A method for stochastic optimization
Original
Diederik P Kingma and Jimmy Ba. 2014 · 2014
Earlier work this paper cites.
Wikidata: a free collaborative knowledgebase
Denny Vrandecic and Markus Krötzsch. 2014 · 2014
Earlier work this paper cites.
Learning Entity and Relation Embeddings for Knowledge Graph Completion. In AAAI . AAAI Press, 2181–2187
Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. 2015 · 2015
Earlier work this paper cites.
Deep Residual Learning for Image Recognition. In CVPR . IEEE Computer Society, 770–778
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016 · 2016
Earlier work this paper cites.
YAGO: A Multilingual Knowledge Base from Wikipedia, Wordnet, and Geonames. In ISWC (2) (Lecture Notes in Computer Science, Vol. 9982) . 177–185
Thomas Rebele, Fabian M. Suchanek, Johannes Hoffart, Joanna Biega, Erdal Kuzey, and Gerhard Weikum. 2016 · 2016
Earlier work this paper cites.
ConceptNet 5.5: An Open Multilingual Graph of General Knowledge. In AAAI . AAAI Press, 4444–4451
Robyn Speer, Joshua Chin, and Catherine Havasi. 2017 · 2017
Earlier work this paper cites.
Attention is All you Need. In NIPS . 5998–6008
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017 · 2017
Earlier work this paper cites.
Improving language understanding by generative pre-training
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018 · 2018
Earlier work this paper cites.
Modeling Relational Data with Graph Convolutional Networks. In ESWC (Lecture Notes in Computer Science, Vol. 10843) . Springer, 593–607
Michael Sejr Schlichtkrull, Thomas N. Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. 2018 · 2018
Earlier work this paper cites.
Bootstrapping Entity Alignment with Knowledge Graph Embedding. In IJCAI . ijcai.org, 4396–4402
Zequn Sun, Wei Hu, Qingheng Zhang, and Yuzhong Qu. 2018 · 2018
Earlier work this paper cites.
Graph attention networks
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2018 · 2018
Earlier work this paper cites.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL-HLT (1) . Association for Computational Linguistics, 4171–4186
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 · 2019
Earlier work this paper cites.
Knowledge Graph Embedding Based Question Answering. In WSDM . ACM, 105–113
Xiao Huang, Jingyuan Zhang, Dingcheng Li, and Ping Li. 2019 · 2019
Earlier work this paper cites.
Text Generation from Knowledge Graphs with Graph Transformers. In NAACL-HLT (1) . Association for Computational Linguistics, 2284–2293
Rik Koncel-Kedziorski, Dhanush Bekal, Yi Luan, Mirella Lapata, and Hannaneh Hajishirzi. 2019 · 2019
Earlier work this paper cites.
KagNet: Knowledge-Aware Graph Networks for Commonsense Reasoning. In EMNLP/IJCNLP (1) . Association for Computational Linguistics, 2829–2839
Bill Yuchen Lin, Xinyue Chen, Jamin Chen, and Xiang Ren. 2019 · 2019
Earlier work this paper cites.
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Original
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019 · 2019
Earlier work this paper cites.
ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks. In NeurIPS . 13–23
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019 · 2019
Earlier work this paper cites.

Similar

RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space. In ICLR (Poster) . OpenReview.net
Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nie, and Jian Tang. 2019 · 2019
Cited alongside, same era.
CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge. In NAACL-HLT (1) . Association for Computational Linguistics, 4149–4158
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019 · 2019
Cited alongside, same era.
Zero-Shot Learning - A Comprehensive Evaluation of the Good, the Bad and the Ugly
Yongqin Xian, Christoph H. Lampert, Bernt Schiele, and Zeynep Akata. 2019 · 2019
Cited alongside, same era.
KG-BERT: BERT for Knowledge Graph Completion
Original
Liang Yao, Chengsheng Mao, and Yuan Luo. 2019 · 2019
Cited alongside, same era.
Iteratively Learning Embeddings and Rules for Knowledge Graph Reasoning. In WWW . ACM, 2366–2377
Wen Zhang, Bibek Paudel, Liang Wang, Jiaoyan Chen, Hai Zhu, Wei Zhang, Abraham Bernstein, and Huajun Chen. 2019 · 2019
Cited alongside, same era.
Scalable Multi-Hop Relational Reasoning for Knowledge-Aware Question Answering. In EMNLP (1) . Association for Computational Linguistics, 1295–1309
Yanlin Feng, Xinyue Chen, Bill Yuchen Lin, Peifeng Wang, Jun Yan, and Xiang Ren. 2020 · 2020
Cited alongside, same era.
FashionBERT: Text and Image Matching with Adaptive Loss for Cross-modal Retrieval. In SIGIR . ACM, 2251–2260
Dehong Gao, Linbo Jin, Ben Chen, Minghui Qiu, Peng Li, Yi Wei, Yi Hu, and Hao Wang. 2020 · 2020
Cited alongside, same era.
ConceptBert: Concept-Aware Representation for Visual Question Answering. In EMNLP (Findings) (Findings of ACL, Vol. EMNLP 2020) . Association for Computational Linguistics, 489–498
François Gardères, Maryam Ziaeefard, Baptiste Abeloos, and Freddy Lécué. 2020 · 2020
Cited alongside, same era.
TaPas: Weakly Supervised Table Parsing via Pre-training. In ACL . Association for Computational Linguistics, 4320–4333
Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Martin Eisenschlos. 2020 · 2020
Cited alongside, same era.
Multi-Task Learning for Knowledge Graph Completion with Pre-trained Language Models. In COLING . International Committee on Computational Linguistics, 1737–1743
Bosung Kim, Taesuk Hong, Youngjoong Ko, and Jungyun Seo. 2020 · 2020
Cited alongside, same era.
BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. In ACL . Association for Computational Linguistics, 7871–7880
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020 · 2020
Cited alongside, same era.
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020 · 2020
Cited alongside, same era.

Then

Making Pre-trained Language Models Better Few-shot Learners. In ACL/IJCNLP (1) . Association for Computational Linguistics, 3816–3830
Tianyu Gao, Adam Fisch, and Danqi Chen. 2021 · 2021
Later among the works it cites.
K-ZSL: resources for knowledge-driven zero-shot learning
Original
Yuxia Geng, Jiaoyan Chen, Zhuo Chen, Jeff Z Pan, Zonggang Yuan, and Huajun Chen. 2021b · 2021
Later among the works it cites.
The Power of Scale for Parameter-Efficient Prompt Tuning. In EMNLP (1) . Association for Computational Linguistics, 3045–3059
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021 · 2021
Later among the works it cites.
Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing
Original
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021 · 2021
Later among the works it cites.
Communicative Message Passing for Inductive Relation Reasoning. In AAAI . AAAI Press, 4294–4302
Sijie Mai, Shuangjia Zheng, Yuedong Yang, and Haifeng Hu. 2021 · 2021
Later among the works it cites.
KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation
Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhengyan Zhang, Zhiyuan Liu, Juanzi Li, and Jian Tang. 2021b · 2021
Later among the works it cites.
QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering. In NAACL-HLT . Association for Computational Linguistics, 535–546
Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, and Jure Leskovec. 2021 · 2021
Later among the works it cites.
Do Transformers Really Perform Badly for Graph Representation?. In NeurIPS . 28877–28888
Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie-Yan Liu. 2021 · 2021
Later among the works it cites.
Billion-scale Pre-trained E-commerce Product Knowledge Graph Model. In ICDE . IEEE, 2476–2487
Wen Zhang, Chi Man Wong, Ganqiang Ye, Bo Wen, Wei Zhang, and Huajun Chen. 2021 · 2021
Later among the works it cites.
BEiT: BERT Pre-Training of Image Transformers. In ICLR . OpenReview.net
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. 2022 · 2022
Later among the works it cites.
Hybrid Transformer with Multi-level Fusion for Multimodal Knowledge Graph Completion. In SIGIR . ACM, 904–915
Xiang Chen, Ningyu Zhang, Lei Li, Shumin Deng, Chuanqi Tan, Changliang Xu, Fei Huang, Luo Si, and Huajun Chen. 2022 · 2022
Later among the works it cites.
Disentangled Ontology Embedding for Zero-shot Learning. In KDD . ACM, 443–453
Yuxia Geng, Jiaoyan Chen, Wen Zhang, Yajing Xu, Zhuo Chen, Jeff Z. Pan, Yufeng Huang, Feiyu Xiong, and Huajun Chen. 2022 · 2022
Later among the works it cites.
Transformer-based Entity Typing in Knowledge Graphs. In EMNLP . Association for Computational Linguistics, 5988–6001
Zhiwei Hu, Víctor Gutiérrez-Basulto, Zhiliang Xiang, Ru Li, and Jeff Z. Pan. 2022 · 2022
Later among the works it cites.
Learning Knowledge Graph Embedding With Heterogeneous Relation Attention Networks
Zhifei Li, Hai Liu, Zhaoli Zhang, Tingting Liu, and Neal N. Xiong. 2022a · 2022
Later among the works it cites.
Incorporating Context Graph with Logical Reasoning for Inductive Relation Prediction. In SIGIR . ACM, 893–903
Qika Lin, Jun Liu, Fangzhi Xu, Yudai Pan, Yifan Zhu, Lingling Zhang, and Tianzhe Zhao. 2022 · 2022
Later among the works it cites.
Sequence-to-Sequence Knowledge Graph Completion and Question Answering. In ACL (1) . Association for Computational Linguistics, 2814–2828
Apoorv Saxena, Adrian Kochsiek, and Rainer Gemulla. 2022 · 2022
Later among the works it cites.
Graph Hawkes Transformer for Extrapolated Reasoning on Temporal Knowledge Graphs. In EMNLP . Association for Computational Linguistics, 7481–7493
Haohai Sun, Shangyi Geng, Jialun Zhong, Han Hu, and Kun He. 2022 · 2022
Later among the works it cites.
Neural-Symbolic Entangled Framework for Complex Query Answering
Original
Zezhong Xu, Wen Zhang, Peng Ye, Hui Chen, and Huajun Chen. 2022b · 2022
Later among the works it cites.
Complex Embeddings for Simple Link Prediction. In ICML (JMLR Workshop and Conference Proceedings, Vol. 48) . JMLR.org, 2071–2080
Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. 2016 · 2080
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…