ManyModalQA: Modality Disambiguation and QA over Diverse Inputs

Understand

We present a new multimodal question answering challenge, ManyModalQA, in which an agent must answer a question by considering three distinct modalities: text, images, and tables.

• We collect our data by scraping Wikipedia and then utilize crowdsourcing to collect question-answer pairs.
• Our questions are ambiguous, in that the modality that contains the answer is not easily determined based solely upon the question.
• To demonstrate this ambiguity, we construct a modality selector (or disambiguator) network, and this model gets substantially lower accuracy on our challenge set, compared to existing datasets, indicating that our questions are more ambiguous.