Unveiling Cross Modality Bias in Visual Question Answering: A Causal View with Possible Worlds VQA

Understand

To increase the generalization capability of VQA systems, many recent studies have tried to de-bias spurious language or vision associations that shortcut the question or image to the answer.

• Despite these efforts, the literature fails to address the confounding effect of vision and language simultaneously.
• As a result, when they reduce bias learned from one modality, they usually increase bias from the other.
• In this paper, we first model a confounding effect that causes language and vision bias simultaneously, then propose a counterfactual inference to remove the influence of this effect.