2021

Language Models As or For Knowledge Bases

Razniewski, Simon, Yates, Andrew, Kassner, Nora et al.

Understand

Pre-trained language models (LMs) have recently gained attention for their potential as an alternative to (or proxy for) explicit knowledge bases (KBs).

In this position paper, we examine this hypothesis, identify strengths and limitations of both LMs and KBs, and discuss the complementary nature of the two paradigms.
In particular, we offer qualitative arguments that latent LMs are not suitable as a substitute for explicit KBs, but could play a major role for augmenting and curating KBs.

Built on

Knowledge graph embedding: A survey of approaches and applications,
Q. Wang, Z. Mao, B. Wang, L. Guo, · 2017
Earlier work this paper cites.
Recent trends in deep learning based natural language processing,
T. Young, D. Hazarika, S. Poria, E. Cambria, · 2018
Earlier work this paper cites.
Object hallucination in image captioning,
A. Rohrbach, et al., · 2018
Earlier work this paper cites.
BERT: Pre-training of deep bidirectional transformers for language understanding,
J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, · 2019
Earlier work this paper cites.
Language models are unsupervised multitask learners,
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, · 2019
Earlier work this paper cites.
Exploring the limits of transfer learning with a unified text-to-text transformer,
C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, P. J. Liu, · 2019
Earlier work this paper cites.
Language models as knowledge bases?,
F. Petroni, T. Rocktäschel, S. Riedel, P. Lewis, A. Bakhtin, Y. Wu, A. Miller, · 2019
Earlier work this paper cites.
Industry-scale knowledge graphs: lessons and challenges,
N. Noy, et al., · 2019
Earlier work this paper cites.
What we talk about when we talk about Wikidata quality: a literature survey,
A. Piscopo, E. Simperl, · 2019
Earlier work this paper cites.
E-BERT: Efficient-yet-effective entity embeddings for BERT,
N. Poerner, U. Waltinger, H. Schütze, · 2020
Earlier work this paper cites.

Similar

How much knowledge can you pack into the parameters of a language model?,
A. Roberts, C. Raffel, N. Shazeer, · 2020
Cited alongside, same era.
How context affects language models’ factual predictions,
F. Petroni, P. Lewis, A. Piktus, T. Rocktäschel, Y. Wu, A. H. Miller, S. Riedel, · 2020
Cited alongside, same era.
Realm: Retrieval-augmented language model pre-training,
K. Guu, K. Lee, Z. Tung, P. Pasupat, M.-W. Chang, · 2020
Cited alongside, same era.
Structured knowledge: Have we made progress? an extrinsic study of KB coverage over 19 years,
S. Razniewski, P. Das, · 2020
Cited alongside, same era.
J. Taylor, Automated
2020
Cited alongside, same era.
On exposure bias, hallucination and domain shift in neural machine translation,
C. Wang, R. Sennrich, · 2020
Cited alongside, same era.
Modifying memories in transformer models,
C. Zhu, A. S. Rawat, M. Zaheer, S. Bhojanapalli, D. Li, F. Yu, S. Kumar, · 2020
Cited alongside, same era.
Entities as experts: Sparse memory access with entity supervision,
T. Févry, L. B. Soares, N. FitzGerald, E. Choi, T. Kwiatkowski, · 2020
Cited alongside, same era.
Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing,
P. Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, G. Neubig, · 2021
Cited alongside, same era.
Language models as knowledge bases: On entity representations, storage capacity, and paraphrased queries,
B. Heinzerling, K. Inui, · 2021
Cited alongside, same era.
Language models are open knowledge graphs,
C. Wang, X. Liu, D. Song,
Cited in the paper.
K-adapter: Infusing knowledge into pre-trained models with adapters,
R. Wang, et al.,
Cited in the paper.

Then

Retrieval-augmented generation for knowledge-intensive NLP tasks,
P. Lewis, et al., · 2021
Closest in time.
Measuring and improving consistency in pretrained language models,
Y. Elazar, N. Kassner, S. Ravfogel, A. Ravichander, E. Hovy, H. Schütze, Y. Goldberg, · 2021
Closest in time.
Knowledgeable or educated guess? revisiting language models as knowledge bases,
B. Cao, H. Lin, X. Han, L. Sun, L. Yan, M. Liao, T. Xue, J. Xu, · 2021
Closest in time.
Machine knowledge: Creation and curation of comprehensive knowledge bases,
G. Weikum, L. Dong, S. Razniewski, F. Suchanek, · 2021
Closest in time.
A study of the quality of Wikidata,
K. Shenoy, F. Ilievski, D. Garijo, D. Schwabe, P. Szekely, · 2021
Closest in time.
On the dangers of stochastic parrots: Can language models be too big?,
E. M. Bender, T. Gebru, A. McMillan-Major, S. Shmitchell, · 2021
Closest in time.
Negative knowledge for open-world Wikidata,
H. Arnaout, S. Razniewski, G. Weikum, J. Z. Pan, · 2021
Closest in time.
Editing factual knowledge in language models,
N. D. Cao, W. Aziz, I. Titov, · 2021
Closest in time.
Adaptable and interpretable neural memory over symbolic knowledge,
H. Sun, L. B. Soares, P. Verga, W. W. Cohen, · 2021
Closest in time.
Enriching a model’s notion of belief using a persistent memory,
N. Kassner, O. Tafjord, H. Schutze, P. Clark, · 2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…