Fetching the paper…

Reading the bibliography…

2024

MedPromptX: Grounded Multimodal Prompting for Chest X-ray Diagnosis

Shaaban, Mai A., Khan, Adnan, Yaqub, Mohammad

Original
PDF
Post on X

Understand

Chest X-ray images are commonly used for predicting acute and chronic cardiopulmonary conditions, but efforts to integrate them with structured clinical data face challenges due to incomplete electronic health records (EHR).

This paper introduces MedPromptX, the first clinical decision support system that integrates multimodal large language models (MLLMs), few-shot prompting (FP) and visual grounding (VG) to combine imagery with EHR data for chest X-ray diagnosis.
A pre-trained MLLM is utilized to complement the missing EHR information, providing a comprehensive understanding of patients' medical history.
Additionally, FP reduces the necessity for extensive training of MLLMs while effectively tackling the issue of hallucination.

MedPromptX: Grounded Multimodal Prompting for Chest X-ray Diagnosis · Around

Built on

Johnson, A.E.W., Pollard, T.J., Greenbaum, N.R., et al.: MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs (jan 2019), https://arxiv.org/abs/1901.07042v5
Original
1901
Earlier work this paper cites.
Brown, T.B., Mann, B., Ryder, N., et al.: Language Models are Few-Shot Learners. Advances in Neural Information Processing Systems 2020-Decem
Original
2005
Earlier work this paper cites.
Devlin, J., Chang, M.W., Lee, K., et al.: Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
Original
2018
Earlier work this paper cites.
Shah, S.M., Khan, R.A.: Secondary use of electronic health record: Opportunities and challenges. IEEE access 8
2020
Earlier work this paper cites.
van Leeuwen, K.G., de Rooij, M., Schalekamp, S., et al.: How does artificial intelligence in radiology improve efficiency and health outcomes? Pediatric Radiology pp. 1–7 (2021)
2021
Earlier work this paper cites.
Liu, Z., Lin, Y., Cao, Y., et al.: Swin transformer: Hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 10012–10022 (2021)
2021
Earlier work this paper cites.
Radford, A., Kim, J.W., Hallacy, C., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PMLR (2021)
2021
Earlier work this paper cites.
Alayrac, J.B., Donahue, J., Luc, P., et al.: Flamingo: a Visual Language Model for Few-Shot Learning. Advances in Neural Information Processing Systems 35
Original
2022
Earlier work this paper cites.
Bolton, E., Hall, D., Yasunaga, M., et al.: Stanford crfm: Biomedlm (2022), https://crfm.stanford.edu/2022/12/15/biomedlm.html
2022
Earlier work this paper cites.

Similar

Li, L.H., Zhang, P., Zhang, H., et al.: Grounded language-image pre-training. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10965–10975 (2022)
2022
Cited alongside, same era.
Lu, Q., Dou, D., Nguyen, T.H.: ClinicalT5: A Generative Language Model for Clinical Text. Findings of the Association for Computational Linguistics: EMNLP 2022 pp. 5436–5443 (2022). https://doi.org/10.18653/V1/2022.FINDINGS-EMNLP.398
2022
Cited alongside, same era.
Soenksen, L.R., Ma, Y., Zeng, C., et al.: Integrated multimodal artificial intelligence framework for healthcare applications. npj Digital Medicine 2022 5:1 5
2022
Cited alongside, same era.
Zhang, H., Li, F., Liu, S., et al.: Dino: Detr with improved denoising anchor boxes for end-to-end object detection. arXiv preprint arXiv:2203.03605 (2022)
Original
2022
Cited alongside, same era.

Then

Moor, M., Huang, Q., Wu, S., et al.: Med-Flamingo: a Multimodal Medical Few-shot Learner (jul 2023), https://arxiv.org/abs/2307.15189v1
Original
2023
Later among the works it cites.
Najjar, R.: Redefining radiology: a review of artificial intelligence integration in medical imaging. Diagnostics 13
2023
Later among the works it cites.
Touvron, H., Lavril, T., Izacard, G., et al.: Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)
Original
2023
Later among the works it cites.
Tu, T., Azizi, S., Driess, D., et al.: Towards Generalist Biomedical AI (jul 2023), https://arxiv.org/abs/2307.14334v1
Original
2023
Later among the works it cites.
Yin, S., Fu, C., Zhao, S., et al.: Woodpecker: Hallucination correction for multimodal large language models (2023). https://doi.org/10.48550/ARXIV.2310.16045

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Awadalla, A., Gao, I., Gardner, J., et al.: Openflamingo: An open-source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.01390 (2023)

2023

Cited alongside, same era.

Ichinose, A., Hatsutani, T., Nakamura, K., et al.: Visual Grounding of Whole Radiology Reports for 3D CT Images, p. 611–621. Springer Nature Switzerland (2023). https://doi.org/10.1007/978-3-031-43904-9_59

2023

Cited alongside, same era.

Johnson, A.E., Bulgarelli, L., Shen, L., et al.: MIMIC-IV, a freely accessible electronic health record dataset. Scientific Data 2023 10:1 10

2023

Cited alongside, same era.

Li, Y., Liu, Y., Wang, Z., et al.: A comprehensive study of gpt-4v’s multimodal capabilities in medical imaging. medRxiv pp. 2023–11 (2023)

2023

Cited alongside, same era.

Liu, S., Zeng, Z., Ren, T., et al.: Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499 (2023)

2023

Cited alongside, same era.

Yang, Z., Gan, Z., Wang, J., et al.: An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA

Cited in the paper.

2023

Later among the works it cites.

Zhou, H., Liu, F., Gu, B., et al.: A Survey of Large Language Models in Medicine: Principles, Applications, and Challenges (nov 2023), https://arxiv.org/abs/2311.05112v2

2023

Later among the works it cites.

Zhang, K., Yu, J., Adhikarla, E., et al.: Biomedgpt: A unified and generalist biomedical generative pre-trained transformer for vision, language, and multimodal tasks (2024)

2024

Closest in time.

Zhu, Y., Wang, Z., Gao, J., Tong, Y., An, J., Liao, W., Harrison, E.M., Ma, L., Pan, C.: Prompting large language models for zero-shot clinical prediction with structured longitudinal electronic health record data. arXiv preprint arXiv:2402.01713 (2024)

2024

Closest in time.