K-BERT: Enabling Language Representation with Knowledge Graph

Understand

Pre-trained language representation models, such as BERT, capture a general language representation from large-scale corpora, but lack domain-specific knowledge.

• When reading a domain text, experts make inferences with relevant knowledge.
• For machines to achieve this capability, we propose a knowledge-enabled language representation model (K-BERT) with knowledge graphs (KGs), in which triples are injected into the sentences as domain knowledge.
• However, too much knowledge incorporation may divert the sentence from its correct meaning, which is called knowledge noise (KN) issue.