PixFoundation: Are We Heading in the Right Direction with Pixel-level Vision Foundation Models?

Understand

Multiple works have emerged to push the boundaries of multi-modal large language models (MLLMs) towards pixel-level understanding.

• The current trend is to train MLLMs with pixel-level grounding supervision in terms of masks on large-scale labelled data and specialized decoders for the segmentation task.
• However, we show that such MLLMs when evaluated on recent challenging vision-centric benchmarks, exhibit a weak ability in visual question answering (VQA).
• Surprisingly, some of these methods even downgrade the grounding ability of MLLMs that were never trained with such pixel-level supervision.