MDNet: A Semantically and Visually Interpretable Medical Image Diagnosis Network

Understand

The inability to interpret the model prediction in semantically and visually meaningful ways is a well-known shortcoming of most existing computer-aided diagnosis methods.

• In this paper, we propose MDNet to establish a direct multimodal mapping between medical images and diagnostic reports that can read images, generate diagnostic reports, retrieve images by symptom descriptions, and visualize attention, to provide justifications of the network diagnosis process.
• MDNet includes an image model and a language model.
• The image model is proposed to enhance multi-scale feature ensembles and utilization efficiency.