P-Adapters: Robustly Extracting Factual Information from Language Models with Diverse Prompts

Understand

Recent work (e.g.

• LAMA (Petroni et al., 2019)) has found that the quality of the factual information extracted from Large Language Models (LLMs) depends on the prompts used to query them.
• This inconsistency is problematic because different users will query LLMs for the same information using different wording, but should receive the same, accurate responses regardless.
• In this work we aim to address this shortcoming by introducing P-Adapters: lightweight models that sit between the embedding layer and first attention layer of LLMs.