Training Multimedia Event Extraction With Generated Images and Captions

Understand

Contemporary news reporting increasingly features multimedia content, motivating research on multimedia event extraction.

• However, the task lacks annotated multimodal training data and artificially generated training data suffer from distribution shift from real-world data.
• In this paper, we propose Cross-modality Augmented Multimedia Event Learning (CAMEL), which successfully utilizes artificially generated multimodal training data and achieves state-of-the-art performance.
• We start with two labeled unimodal datasets in text and image respectively, and generate the missing modality using off-the-shelf image generators like Stable Diffusion and image captioners like BLIP.