A Survey of Knowledge Enhanced Pre-trained Models

Understand

Pre-trained language models learn informative word representations on a large-scale text corpus through self-supervised learning, which has achieved promising performance in fields of natural language processing (NLP) after fine-tuning.

• These models, however, suffer from poor robustness and lack of interpretability.
• We refer to pre-trained language models with knowledge injection as knowledge-enhanced pre-trained language models (KEPLMs).
• These models demonstrate deep understanding and logical reasoning and introduce interpretability.