On the Generalization Ability of Retrieval-Enhanced Transformers

Understand

Recent work on the Retrieval-Enhanced Transformer (RETRO) model has shown that off-loading memory from trainable weights to a retrieval database can significantly improve language modeling and match the performance of non-retrieval models that are an order of magnitude larger in size.

• It has been suggested that at least some of this performance gain is due to non-trivial generalization based on both model weights and retrieval.
• In this paper, we try to better understand the relative contributions of these two components.
• We find that the performance gains from retrieval largely originate from overlapping tokens between the database and the test data, suggesting less non-trivial generalization than previously assumed.