VISTA: A Visual and Textual Attention Dataset for Interpreting Multimodal Models

Understand

The recent developments in deep learning led to the integration of natural language processing (NLP) with computer vision, resulting in powerful integrated Vision and Language Models (VLMs).

• Despite their remarkable capabilities, these models are frequently regarded as black boxes within the machine learning research community.
• This raises a critical question: which parts of an image correspond to specific segments of text, and how can we decipher these associations? Understanding these connections is essential for enhancing model transparency, interpretability, and trustworthiness.
• To answer this question, we present an image-text aligned human visual attention dataset that maps specific associations between image regions and corresponding text segments.