PR-MCS: Perturbation Robust Metric for MultiLingual Image Captioning

Understand

Vulnerability to lexical perturbation is a critical weakness of automatic evaluation metrics for image captioning.

• This paper proposes Perturbation Robust Multi-Lingual CLIPScore(PR-MCS), which exhibits robustness to such perturbations, as a novel reference-free image captioning metric applicable to multiple languages.
• To achieve perturbation robustness, we fine-tune the text encoder of CLIP with our language-agnostic method to distinguish the perturbed text from the original text.
• To verify the robustness of PR-MCS, we introduce a new fine-grained evaluation dataset consisting of detailed captions, critical objects, and the relationships between the objects for 3, 000 images in five languages.