Fetching the paper…

Reading the bibliography…

2020

JAKET: Joint Pre-training of Knowledge Graph and Language Understanding

Yu, Donghan, Zhu, Chenguang, Yang, Yiming et al.

Original
PDF
Post on X

Understand

Knowledge graphs (KGs) contain rich information about world knowledge, entities and relations.

Thus, they can be great supplements to existing pre-trained language models.
However, it remains a challenge to efficiently integrate information from KG into language modeling.
And the understanding of a knowledge graph requires related context.

JAKET: Joint Pre-training of Knowledge Graph and Language Understanding · Around

Built on

Linguistic knowledge and transferability of contextual representations
Original
Nelson F Liu, Matt Gardner, Yonatan Belinkov, Matthew E Peters, and Noah A Smith · 1903
Earlier work this paper cites.
Pullnet: Open domain question answering with iterative retrieval on knowledge bases and text
Original
Haitian Sun, Tania Bedrax-Weiss, and William W Cohen · 1904
Earlier work this paper cites.
Ernie: Enhanced representation through knowledge integration
Original
Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu · 1904
Earlier work this paper cites.
Roberta: A robustly optimized bert pretraining approach
Original
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov · 1907
Earlier work this paper cites.
Deep graph library: Towards efficient and scalable deep learning on graphs
Original
Minjie Wang, Lingfan Yu, Da Zheng, Quan Gan, Yu Gai, Zihao Ye, Mufei Li, Jinjing Zhou, Qi Huang, Chao Ma, et al · 1909
Earlier work this paper cites.
Kepler: A unified model for knowledge embedding and pre-trained language representation
Original
Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhiyuan Liu, Juanzi Li, and Jian Tang · 1911
Earlier work this paper cites.
Translating embeddings for modeling multi-relational data
Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko · 2013
Earlier work this paper cites.
Wikidata: a free collaborative knowledgebase
Denny Vrandečić and Markus Krötzsch · 2014
Earlier work this paper cites.
Layer normalization
Original
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton · 2016
Earlier work this paper cites.
Inductive representation learning on large graphs
Will Hamilton, Zhitao Ying, and Jure Leskovec · 2017
Earlier work this paper cites.
Decoupled weight decay regularization
Original
Ilya Loshchilov and Frank Hutter · 2017
Earlier work this paper cites.
Sling: A framework for frame semantic parsing
Original
Michael Ringgaard, Rahul Gupta, and Fernando CN Pereira · 2017
Earlier work this paper cites.
Graph attention networks
Original
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio · 2017
Earlier work this paper cites.

Similar

Variational reasoning for question answering with knowledge graph
Original
Yuyu Zhang, Hanjun Dai, Zornitsa Kozareva, Alexander J Smola, and Le Song · 2017
Cited alongside, same era.
Bert: Pre-training of deep bidirectional transformers for language understanding
Original
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova · 2018
Cited alongside, same era.
Fewrel: A large-scale supervised few-shot relation classification dataset with state-of-the-art evaluation
Original
Xu Han, Hao Zhu, Pengfei Yu, Ziyun Wang, Yuan Yao, Zhiyuan Liu, and Maosong Sun · 2018
Cited alongside, same era.
Deep contextualized word representations
Original
Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer · 2018
Cited alongside, same era.

Then

olmpics–on what language model pre-training captures
Original
Alon Talmor, Yanai Elazar, Yoav Goldberg, and Jonathan Berant · 2019
Later among the works it cites.
Composition-based multi-relational graph convolutional networks
Original
Shikhar Vashishth, Soumya Sanyal, Vikram Nitin, and Partha Talukdar · 2019
Later among the works it cites.
Huggingface’s transformers: State-of-the-art natural language processing
Original
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush · 2019
Later among the works it cites.
Pretrained encyclopedia: Weakly supervised knowledge-pretrained language model
Original
Wenhan Xiong, Jingfei Du, William Yang Wang, and Veselin Stoyanov · 2019
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Improving language understanding by generative pre-training, 2018

Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever · 2018

Cited alongside, same era.

Open domain question answering using early fusion of knowledge bases and text

Haitian Sun, Bhuwan Dhingra, Manzil Zaheer, Kathryn Mazaitis, Ruslan Salakhutdinov, and William W Cohen · 2018

Cited alongside, same era.

Cognitive graph for multi-hop reading comprehension at scale

Ming Ding, Chang Zhou, Qibin Chen, Hongxia Yang, and Jie Tang · 2019

Cited alongside, same era.

Fewrel 2.0: Towards more challenging few-shot relation classification

Tianyu Gao, Xu Han, Hao Zhu, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou · 2019

Cited alongside, same era.

Sensebert: Driving some sense into bert

Yoav Levine, Barak Lenz, Or Dagan, Dan Padnos, Or Sharir, Shai Shalev-Shwartz, Amnon Shashua, and Yoav Shoham · 2019

Cited alongside, same era.

Knowledge enhanced contextual word representations

Matthew E Peters, Mark Neumann, Robert L Logan IV, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A Smith · 2019

Cited alongside, same era.

Bert is not a knowledge base (yet): Factual knowledge vs. name-based reasoning in unsupervised qa

Original

Nina Poerner, Ulli Waltinger, and Hinrich Schütze · 2019

Cited alongside, same era.

Language models are unsupervised multitask learners

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever · 2019

Cited alongside, same era.

Xlnet: Generalized autoregressive pretraining for language understanding

Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le · 2019

Later among the works it cites.

Ernie: Enhanced language representation with informative entities

Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu · 2019

Later among the works it cites.

Language models are few-shot learners

Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al · 2020

Closest in time.

Entities as experts: Sparse memory access with entity supervision

Thibault Févry, Livio Baldini Soares, Nicholas FitzGerald, Eunsol Choi, and Tom Kwiatkowski · 2020

Closest in time.

Realm: Retrieval-augmented language model pre-training

Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang · 2020

Closest in time.

K-bert: Enabling language representation with knowledge graph

Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Qi Ju, Haotang Deng, and Ping Wang · 2020

Closest in time.

Graph-based reasoning over heterogeneous external knowledge for commonsense question answering

Shangwen Lv, Daya Guo, Jingjing Xu, Duyu Tang, Nan Duan, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, and Songlin Hu · 2020

Closest in time.

Exploiting structured knowledge in text via graph-guided representation learning

Tao Shen, Yi Mao, Pengcheng He, Guodong Long, Adam Trischler, and Weizhu Chen · 2020

Closest in time.

Facts as experts: Adaptable and interpretable neural memory over symbolic knowledge

Original

Pat Verga, Haitian Sun, Livio Baldini Soares, and William W Cohen · 2020

Closest in time.