Contextualized Representations Using Textual Encyclopedic Knowledge

Understand

We present a method to represent input texts by contextualizing them jointly with dynamically retrieved textual encyclopedic background knowledge from multiple documents.

• We apply our method to reading comprehension tasks by encoding questions and passages together with background sentences about the entities they mention.
• We show that integrating background knowledge from text is effective for tasks focusing on factual reasoning and allows direct reuse of powerful pretrained BERT-style encoders.
• Moreover, knowledge integration can be further improved with suitable pretraining via a self-supervised masked language model objective over words in background-augmented input text.