Nullu: Mitigating Object Hallucinations in Large Vision-Language Models via HalluSpace Projection

Understand

Recent studies have shown that large vision-language models (LVLMs) often suffer from the issue of object hallucinations (OH).

• To mitigate this issue, we introduce an efficient method that edits the model weights based on an unsafe subspace, which we call HalluSpace in this paper.
• With truthful and hallucinated text prompts accompanying the visual content as inputs, the HalluSpace can be identified by extracting the hallucinated embedding features and removing the truthful representations in LVLMs.
• By orthogonalizing the model weights, input features will be projected into the Null space of the HalluSpace to reduce OH, based on which we name our method Nullu.