Inducing Relational Knowledge from BERT

Understand

One of the most remarkable properties of word embeddings is the fact that they capture certain types of semantic and syntactic relationships.

• Recently, pre-trained language models such as BERT have achieved groundbreaking results across a wide range of Natural Language Processing tasks.
• However, it is unclear to what extent such models capture relational knowledge beyond what is already captured by standard word embeddings.
• To explore this question, we propose a methodology for distilling relational knowledge from a pre-trained language model.