Language Modeling with Editable External Knowledge

Understand

When the world changes, so does the text that humans write about it.

• How do we build language models that can be easily updated to reflect these changes? One popular approach is retrieval-augmented generation, in which new documents are inserted into a knowledge base and retrieved during prediction for downstream tasks.
• Most prior work on these systems have focused on improving behavior during prediction through better retrieval or reasoning.
• This paper introduces ERASE, which instead improves model behavior when new documents are acquired, by incrementally deleting or rewriting other entries in the knowledge base each time a document is added.