Can images help recognize entities? A study of the role of images for Multimodal NER

Understand

Multimodal named entity recognition (MNER) requires to bridge the gap between language understanding and visual context.

• While many multimodal neural techniques have been proposed to incorporate images into the MNER task, the model's ability to leverage multimodal interactions remains poorly understood.
• In this work, we conduct in-depth analyses of existing multimodal fusion techniques from different perspectives and describe the scenarios where adding information from the image does not always boost performance.
• We also study the use of captions as a way to enrich the context for MNER.