IllusionVQA: A Challenging Optical Illusion Dataset for Vision Language Models

Understand

The advent of Vision Language Models (VLM) has allowed researchers to investigate the visual understanding of a neural network using natural language.

• Beyond object classification and detection, VLMs are capable of visual comprehension and common-sense reasoning.
• This naturally led to the question: How do VLMs respond when the image itself is inherently unreasonable? To this end, we present IllusionVQA: a diverse dataset of challenging optical illusions and hard-to-interpret scenes to test the capability of VLMs in two distinct multiple-choice VQA tasks - comprehension and soft localization.
• GPT4V, the best performing VLM, achieves 62.99% accuracy (4-shot) on the comprehension task and 49.7% on the localization task (4-shot and Chain-of-Thought).