A Survey of Knowledge-Intensive NLP with Pre-Trained Language Models

Understand

With the increasing of model capacity brought by pre-trained language models, there emerges boosting needs for more knowledgeable natural language processing (NLP) models with advanced functionalities including providing and making flexible use of encyclopedic and commonsense knowledge.

• The mere pre-trained language models, however, lack the capacity of handling such knowledge-intensive NLP tasks alone.
• To address this challenge, large numbers of pre-trained language models augmented with external knowledge sources are proposed and in rapid development.
• In this paper, we aim to summarize the current progress of pre-trained language model-based knowledge-enhanced models (PLMKEs) by dissecting their three vital elements: knowledge sources, knowledge-intensive NLP tasks, and knowledge fusion methods.