CXR-LLAVA: a multimodal large language model for interpreting chest X-ray images

Understand

Purpose: This study aimed to develop an open-source multimodal large language model (CXR-LLAVA) for interpreting chest X-ray images (CXRs), leveraging recent advances in large language models (LLMs) to potentially replicate the image interpretation skills of human radiologists Materials and Methods: For training, we collected 592,580 publicly available CXRs, of which 374,881 had labels for certain radiographic abnormalities (Dataset 1) and 217,699 provided free-text radiology reports (Dataset 2).

• After pre-training a vision transformer with Dataset 1, we integrated it with an LLM influenced by the LLAVA network.
• Then, the model was fine-tuned, primarily using Dataset 2.
• The model's diagnostic performance for major pathological findings was evaluated, along with the acceptability of radiologic reports by human radiologists, to gauge its potential for autonomous reporting.