2022

Language Models as Knowledge Embeddings

Wang, Xintao, He, Qianyu, Liang, Jiaqing et al.

Understand

Knowledge embeddings (KE) represent a knowledge graph (KG) by embedding entities and relations into continuous vector spaces.

Existing methods are mainly structure-based or description-based.
Structure-based methods learn representations that preserve the inherent structure of KGs.
They cannot well represent abundant long-tail entities in real-world KGs with limited structural information.

Built on

Wordnet: a lexical database for english
George A Miller · 1995
Earlier work this paper cites.
A three-way model for collective learning on multi-relational data
Maximilian Nickel, Volker Tresp, and Hans-Peter Kriegel · 2011
Earlier work this paper cites.
Translating embeddings for modeling multi-relational data
Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko · 2013
Earlier work this paper cites.
Reasoning with neural tensor networks for knowledge base completion
Richard Socher, Danqi Chen, Christopher D Manning, and Andrew Ng · 2013
Earlier work this paper cites.
Knowledge graph embedding by translating on hyperplanes
Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen · 2014
Earlier work this paper cites.
Embedding entities and relations for learning and inference in knowledge bases
Original
Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng · 2014
Earlier work this paper cites.
Learning entity and relation embeddings for knowledge graph completion
Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu · 2015
Earlier work this paper cites.
Observed versus latent features for knowledge base and text inference
Kristina Toutanova · 2015
Earlier work this paper cites.
Complex embeddings for simple link prediction
Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard · 2016
Earlier work this paper cites.

Similar

Representation learning of knowledge graphs with entity descriptions
Ruobing Xie, Zhiyuan Liu, Jia Jia, Huanbo Luan, and Maosong Sun · 2016
Cited alongside, same era.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin · 2017
Cited alongside, same era.
Knowledge graph embedding: A survey of approaches and applications
Quan Wang, Zhendong Mao, Bin Wang, and Li Guo · 2017
Cited alongside, same era.
Convolutional 2d knowledge graph embeddings
Tim Dettmers, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel · 2018
Cited alongside, same era.
Bert: Pre-training of deep bidirectional transformers for language understanding
Original
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova · 2018
Cited alongside, same era.
Tucker: Tensor factorization for knowledge graph completion
Original
Ivana Balažević, Carl Allen, and Timothy M Hospedales · 2019
Cited alongside, same era.
Semi-supervised entity alignment via knowledge graph embedding with awareness of degree difference
Shichao Pei, Lu Yu, Robert Hoehndorf, and Xiangliang Zhang · 2019
Cited alongside, same era.
Language models as knowledge bases?
Original
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel · 2019
Cited alongside, same era.
Rotate: Knowledge graph embedding by relational rotation in complex space
Original
Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nie, and Jian Tang · 2019
Cited alongside, same era.

Then

Coke: Contextualized knowledge graph embedding
Original
Quan Wang, Pingping Huang, Haifeng Wang, Songtai Dai, Wenbin Jiang, Jing Liu, Yajuan Lyu, Yong Zhu, and Hua Wu · 2019
Later among the works it cites.
Kepler: A unified model for knowledge embedding and pre-trained language representation
Original
Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhiyuan Liu, Juanzi Li, and Jian Tang · 2019
Later among the works it cites.
Kg-bert: Bert for knowledge graph completion
Original
Liang Yao, Chengsheng Mao, and Yuan Luo · 2019
Later among the works it cites.
Are pretrained language models symbolic reasoners over knowledge?
Nora Kassner, Benno Krojer, and Hinrich Schütze · 2020
Later among the works it cites.
Structure-augmented text representation learning for efficient knowledge graph completion
Original
Bo Wang, Tao Shen, Guodong Long, Tianyi Zhou, and Yi Chang · 2020
Later among the works it cites.
Learning hierarchy-aware knowledge graph embeddings for link prediction
Zhanqiu Zhang, Jianyu Cai, Yongdong Zhang, and Jie Wang · 2020
Later among the works it cites.
Pretrain-kge: Learning knowledge representation from pretrained language models
Zhiyuan Zhang, Xiaoqian Liu, Yi Zhang, Qi Su, Xu Sun, and Bin He · 2020
Later among the works it cites.
Mem-kgc: Masked entity model for knowledge graph completion with pre-trained language model
Bonggeun Choi, Daesik Jang, and Youngjoong Ko · 2021
Later among the works it cites.
Triple classification for scholarly knowledge graph completion
Original
Mohamad Yaser Jaradeh, Kuldeep Singh, Markus Stocker, and Sören Auer · 2021
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…