LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention

Understand

Entity representations are useful in natural language tasks involving entities.

• In this paper, we propose new pretrained contextualized representations of words and entities based on the bidirectional transformer.
• The proposed model treats words and entities in a given text as independent tokens, and outputs contextualized representations of them.
• Our model is trained using a new pretraining task based on the masked language model of BERT.