Joint Multimodal Entity-Relation Extraction Based on Edge-enhanced Graph Alignment Network and Word-pair Relation Tagging

Understand

Multimodal named entity recognition (MNER) and multimodal relation extraction (MRE) are two fundamental subtasks in the multimodal knowledge graph construction task.

• However, the existing methods usually handle two tasks independently, which ignores the bidirectional interaction between them.
• This paper is the first to propose jointly performing MNER and MRE as a joint multimodal entity-relation extraction task (JMERE).
• Besides, the current MNER and MRE models only consider aligning the visual objects with textual entities in visual and textual graphs but ignore the entity-entity relationships and object-object relationships.