2022

Neural Knowledge Bank for Pretrained Transformers

Dai, Damai, Jiang, Wenbin, Dong, Qingxiu et al.

Understand

The ability of pretrained Transformers to remember factual knowledge is essential but still limited for existing models.

Inspired by existing work that regards Feed-Forward Networks (FFNs) in Transformers as key-value memories, we design a Neural Knowledge Bank (NKB) and a knowledge injection strategy to introduce extra factual knowledge for pretrained Transformers.
The NKB is in the form of additional knowledgeable memory slots to the FFN and the memory-like architecture makes it highly interpretable and flexible.
When injecting extra knowledge with the Salient Span Masking (SSM) pretraining objective, we fix the original pretrained model and train only the NKB.

Built on

RoBERTa: A Robustly Optimized BERT Pretraining Approach
Original
Liu, Y.; Ott, M.; Goyal, N.; Du, J.; Joshi, M.; Chen, D.; Levy, O.; Lewis, M.; Zettlemoyer, L.; and Stoyanov, V. 2019 · 1907
Earlier work this paper cites.
Knowledge Guided Text Retrieval and Reading for Open Domain Question Answering
Original
Min, S.; Chen, D.; Zettlemoyer, L.; and Hajishirzi, H. 2019b · 1911
Earlier work this paper cites.
BERT is Not a Knowledge Base (Yet): Factual Knowledge vs. Name-Based Reasoning in Unsupervised QA
Original
Pörner, N.; Waltinger, U.; and Schütze, H. 2019 · 1911
Earlier work this paper cites.
REALM: Retrieval-Augmented Language Model Pre-Training
Original
Guu, K.; Lee, K.; Tung, Z.; Pasupat, P.; and Chang, M. 2020 · 2002
Earlier work this paper cites.
Rouge: A package for automatic evaluation of summaries
Lin, C.-Y. 2004 · 2004
Earlier work this paper cites.
Open-Domain Question-Answering
Prager, J. M. 2006 · 2006
Earlier work this paper cites.
Deep Sparse Rectifier Neural Networks
Glorot, X.; Bordes, A.; and Bengio, Y. 2011 · 2011
Earlier work this paper cites.
Transformer Feed-Forward Layers Are Key-Value Memories
Original
Geva, M.; Schuster, R.; Berant, J.; and Levy, O. 2020 · 2012
Earlier work this paper cites.
Semantic Parsing on Freebase from Question-Answer Pairs
Berant, J.; Chou, A.; Frostig, R.; and Liang, P. 2013 · 2013
Earlier work this paper cites.
Gaussian Error Linear Units (GELUs)
Original
Hendrycks, D.; and Gimpel, K. 2016 · 2016
Earlier work this paper cites.
Reading Wikipedia to Answer Open-Domain Questions
Chen, D.; Fisch, A.; Weston, J.; and Bordes, A. 2017 · 2017
Earlier work this paper cites.
Fixing Weight Decay Regularization in Adam
Original
Loshchilov, I.; and Hutter, F. 2017 · 2017
Earlier work this paper cites.
Attention is All you Need
Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, L.; and Polosukhin, I. 2017 · 2017
Earlier work this paper cites.

Similar

Don’t Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization
Narayan, S.; Cohen, S. B.; and Lapata, M. 2018 · 2018
Cited alongside, same era.
A Call for Clarity in Reporting BLEU Scores
Post, M. 2018 · 2018
Cited alongside, same era.
Adafactor: Adaptive Learning Rates with Sublinear Memory Cost
Shazeer, N.; and Stern, M. 2018 · 2018
Cited alongside, same era.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J.; Chang, M.; Lee, K.; and Toutanova, K. 2019 · 2019
Cited alongside, same era.
Unified Language Model Pre-training for Natural Language Understanding and Generation
Dong, L.; Yang, N.; Wang, W.; Wei, F.; Liu, X.; Wang, Y.; Gao, J.; Zhou, M.; and Hon, H. 2019 · 2019
Cited alongside, same era.
Natural Questions: a Benchmark for Question Answering Research
Kwiatkowski, T.; Palomaki, J.; Redfield, O.; Collins, M.; Parikh, A. P.; Alberti, C.; Epstein, D.; Polosukhin, I.; Devlin, J.; Lee, K.; Toutanova, K.; Jones, L.; Kelcey, M.; Chang, M.; Dai, A. M.; Uszkoreit, J.; Le, Q.; and Petrov, S. 2019 · 2019
Cited alongside, same era.
Latent Retrieval for Weakly Supervised Open Domain Question Answering
Lee, K.; Chang, M.; and Toutanova, K. 2019 · 2019
Cited alongside, same era.
A Discrete Hard EM Approach for Weakly Supervised Question Answering
Min, S.; Chen, D.; Hajishirzi, H.; and Zettlemoyer, L. 2019a · 2019
Cited alongside, same era.
Knowledge Enhanced Contextual Word Representations
Peters, M. E.; Neumann, M.; IV, R. L. L.; Schwartz, R.; Joshi, V.; Singh, S.; and Smith, N. A. 2019 · 2019
Cited alongside, same era.
Language Models as Knowledge Bases?
Petroni, F.; Rocktäschel, T.; Riedel, S.; Lewis, P. S. H.; Bakhtin, A.; Wu, Y.; and Miller, A. H. 2019 · 2019
Cited alongside, same era.
HEAD-QA: A Healthcare Dataset for Complex Reasoning
Vilares, D.; and Gómez-Rodríguez, C. 2019 · 2019
Cited alongside, same era.
Pay Less Attention with Lightweight and Dynamic Convolutions
Wu, F.; Fan, A.; Baevski, A.; Dauphin, Y. N.; and Auli, M. 2019 · 2019
Cited alongside, same era.

Then

ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators
Clark, K.; Luong, M.; Le, Q. V.; and Manning, C. D. 2020 · 2020
Later among the works it cites.
How Can We Know What Language Models Know?
Jiang, Z.; Xu, F. F.; Araki, J.; and Neubig, G. 2020 · 2020
Later among the works it cites.
K-BERT: Enabling Language Representation with Knowledge Graph
Liu, W.; Zhou, P.; Zhao, Z.; Wang, Z.; Ju, Q.; Deng, H.; and Wang, P. 2020 · 2020
Later among the works it cites.
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C.; Shazeer, N.; Roberts, A.; Lee, K.; Narang, S.; Matena, M.; Zhou, Y.; Li, W.; and Liu, P. J. 2020 · 2020
Later among the works it cites.
How Much Knowledge Can You Pack Into the Parameters of a Language Model?
Roberts, A.; Raffel, C.; and Shazeer, N. 2020 · 2020
Later among the works it cites.
Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Bases
Cao, B.; Lin, H.; Han, X.; Sun, L.; Yan, L.; Liao, M.; Xue, T.; and Xu, J. 2021 · 2021
Later among the works it cites.
Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth
Original
Dong, Y.; Cordonnier, J.; and Loukas, A. 2021 · 2021
Later among the works it cites.
Measuring and Improving Consistency in Pretrained Language Models
Original
Elazar, Y.; Kassner, N.; Ravfogel, S.; Ravichander, A.; Hovy, E. H.; Schütze, H.; and Goldberg, Y. 2021 · 2021
Later among the works it cites.
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
Wang, R.; Tang, D.; Duan, N.; Wei, Z.; Huang, X.; Ji, J.; Cao, G.; Jiang, D.; and Zhou, M. 2021a · 2021
Later among the works it cites.
Knowledge Neurons in Pretrained Transformers
Dai, D.; Dong, L.; Hao, Y.; Sui, Z.; Chang, B.; and Wei, F. 2022 · 2022
Closest in time.
Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space
Original
Geva, M.; Caciularu, A.; Wang, K. R.; and Goldberg, Y. 2022 · 2022
Closest in time.
Kformer: Knowledge Injection in Transformer Feed-Forward Layers
Original
Yao, Y.; Huang, S.; Zhang, N.; Dong, L.; Wei, F.; and Chen, H. 2022 · 2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…