Med3DVLM: An Efficient Vision-Language Model for 3D Medical Image Analysis

Understand

Vision-language models (VLMs) have shown promise in 2D medical image analysis, but extending them to 3D remains challenging due to the high computational demands of volumetric data and the difficulty of aligning 3D spatial features with clinical text.

• We present Med3DVLM, a 3D VLM designed to address these challenges through three key innovations: (1) DCFormer, an efficient encoder that uses decomposed 3D convolutions to capture fine-grained spatial features at scale; (2) SigLIP, a contrastive learning strategy with pairwise sigmoid loss that improves image-text alignment without relying on large negative batches; and (3) a dual-stream MLP-Mixer projector that fuses low- and high-level image features with text embeddings for richer multi-modal representations.
• We evaluate our model on the M3D dataset, which includes radiology reports and VQA data for 120,084 3D medical images.
• Results show that Med3DVLM achieves superior performance across multiple benchmarks.