Fine-grained and Explainable Factuality Evaluation for Multimodal Summarization

Understand

Multimodal summarization aims to generate a concise summary based on the input text and image.

• However, the existing methods potentially suffer from unfactual output.
• To evaluate the factuality of multimodal summarization models, we propose two fine-grained and explainable evaluation frameworks (FALLACIOUS) for different application scenarios, i.e.
• reference-based factuality evaluation framework and reference-free factuality evaluation framework.