Multimodal Explanations: Justifying Decisions and Pointing to the Evidence

Understand

Deep models that are both effective and explainable are desirable in many settings; prior explainable models have been unimodal, offering either image-based visualization of attention weights or text-based generation of post-hoc justifications.

• We propose a multimodal approach to explanation, and argue that the two modalities provide complementary explanatory strengths.
• We collect two new datasets to define and evaluate this task, and propose a novel model which can provide joint textual rationale generation and attention visualization.
• Our datasets define visual and textual justifications of a classification decision for activity recognition tasks (ACT-X) and for visual question answering tasks (VQA-X).