Enhancing Explainability in Multimodal Large Language Models Using Ontological Context

Understand

Recently, there has been a growing interest in Multimodal Large Language Models (MLLMs) due to their remarkable potential in various tasks integrating different modalities, such as image and text, as well as applications such as image captioning and visual question answering.

• However, such models still face challenges in accurately captioning and interpreting specific visual concepts and classes, particularly in domain-specific applications.
• We argue that integrating domain knowledge in the form of an ontology can significantly address these issues.
• In this work, as a proof of concept, we propose a new framework that combines ontology with MLLMs to classify images of plant diseases.