Significantly improving zero-shot X-ray pathology classification via fine-tuning pre-trained image-text encoders

Understand

Deep neural networks are increasingly used in medical imaging for tasks such as pathological classification, but they face challenges due to the scarcity of high-quality, expert-labeled training data.

• Recent efforts have utilized pre-trained contrastive image-text models like CLIP, adapting them for medical use by fine-tuning the model with chest X-ray images and corresponding reports for zero-shot pathology classification, thus eliminating the need for pathology-specific annotations.
• However, most studies continue to use the same contrastive learning objectives as in the general domain, overlooking the multi-labeled nature of medical image-report pairs.
• In this paper, we propose a new fine-tuning strategy that includes positive-pair loss relaxation and random sentence sampling.