A Causal View of Entity Bias in (Large) Language Models

Understand

Entity bias widely affects pretrained (large) language models, causing them to rely on (biased) parametric knowledge to make unfaithful predictions.

• Although causality-inspired methods have shown great potential to mitigate entity bias, it is hard to precisely estimate the parameters of underlying causal models in practice.
• The rise of black-box LLMs also makes the situation even worse, because of their inaccessible parameters and uncalibrated logits.
• To address these problems, we propose a specific structured causal model (SCM) whose parameters are comparatively easier to estimate.