Fetching the paper…

Reading the bibliography…

2023

IMITATE: Clinical Prior Guided Hierarchical Vision-Language Pre-training

Liu, Che, Cheng, Sibo, Shi, Miaojing et al.

Original
PDF
Post on X

Understand

In the field of medical Vision-Language Pre-training (VLP), significant efforts have been devoted to deriving text and image features from both clinical reports and associated medical images.

However, most existing methods may have overlooked the opportunity in leveraging the inherent hierarchical structure of clinical reports, which are generally split into `findings' for descriptive content and `impressions' for conclusive observation.
Instead of utilizing this rich, structured format, current medical VLP approaches often simplify the report into either a unified entity or fragmented tokens.
In this work, we propose a novel clinical prior guided VLP framework named IMITATE to learn the structure information from medical reports with hierarchical vision-language alignment.

IMITATE: Clinical Prior Guided Hierarchical Vision-Language Pre-training · Around

Built on

J. MacQueen et al. , “Some methods for classification and analysis of multivariate observations,” in Proceedings of the fifth Berkeley symposium on mathematical statistics and probability , vol. 1, no. 14. Oakland, CA, USA, 1967, pp. 281–297
1967
Earlier work this paper cites.
L. Van der Maaten and G. Hinton, “Visualizing data using t-sne.” Journal of machine learning research , vol. 9, no. 11, 2008
2008
Earlier work this paper cites.
D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980 , 2014
Original
2014
Earlier work this paper cites.
O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18 . Springer, 2015, pp. 234–241
2015
Earlier work this paper cites.
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 770–778
2016
Earlier work this paper cites.
A. E. Johnson et al. , “Mimic-iii, a freely accessible critical care database,” Scientific data , vol. 3, no. 1, pp. 1–9, 2016
2016
Earlier work this paper cites.
I. Loshchilov and F. Hutter, “Sgdr: Stochastic gradient descent with warm restarts,” arXiv preprint arXiv:1608.03983 , 2016
Original
2016
Earlier work this paper cites.
A. Vaswani et al. , “Attention is all you need,” Advances in neural information processing systems , vol. 30, 2017
2017
Earlier work this paper cites.
R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-cam: Visual explanations from deep networks via gradient-based localization,” in Proceedings of the IEEE international conference on computer vision , 2017, pp. 618–626
2017
Earlier work this paper cites.
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805 , 2018
Original
2018
Earlier work this paper cites.
T. Baltrušaitis, C. Ahuja, and L.-P. Morency, “Multimodal machine learning: A survey and taxonomy,” IEEE transactions on pattern analysis and machine intelligence , vol. 41, no. 2, pp. 423–443, 2018
2018
Earlier work this paper cites.
J. Redmon and A. Farhadi, “Yolov3: An incremental improvement,” arXiv preprint arXiv:1804.02767 , 2018
Original
2018
Earlier work this paper cites.
M. Z. Hossain, F. Sohel, M. F. Shiratuddin, and H. Laga, “A comprehensive survey of deep learning for image captioning,” ACM Computing Surveys (CsUR) , vol. 51, no. 6, pp. 1–36, 2019
2019
Earlier work this paper cites.
J. Lu, D. Batra, D. Parikh, and S. Lee, “Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks,” Advances in neural information processing systems , vol. 32, 2019
2019
Earlier work this paper cites.
E. Alsentzer et al. , “Publicly available clinical bert embeddings,” arXiv preprint arXiv:1904.03323 , 2019
Original
2019
Earlier work this paper cites.
A. E. Johnson et al. , “Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports,” Scientific data , vol. 6, no. 1, pp. 1–8, 2019
2019
Earlier work this paper cites.
A. E. Johnson et al. , “Mimic-cxr-jpg, a large publicly available database of labeled chest radiographs,” arXiv preprint arXiv:1901.07042 , 2019
Original
2019
Earlier work this paper cites.
J. Irvin et al. , “Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison,” in Proceedings of the AAAI conference on artificial intelligence , vol. 33, 2019, pp. 590–597
2019
Earlier work this paper cites.
G. Shih et al. , “Augmenting the national institutes of health chest radiograph dataset with expert annotations of possible pneumonia,” Radiology: Artificial Intelligence , vol. 1, no. 1, p. e180041, 2019
2019
Earlier work this paper cites.

Similar

C. Steven G. Langer, PhD and M. George Shih, MD, “Siim-acr pneumothorax segmentation,” 2019
2019
Cited alongside, same era.
K. Huang, J. Altosaar, and R. Ranganath, “Clinicalbert: Modeling clinical notes and predicting hospital readmission,” arXiv preprint arXiv:1904.05342 , 2019
Original
2019
Cited alongside, same era.
Y. Zhang, H. Jiang, Y. Miura, C. D. Manning, and C. P. Langlotz, “Contrastive learning of medical visual representations from paired images and text,” arXiv preprint arXiv:2010.00747 , 2020
Original
2020
Cited alongside, same era.
A. Dosovitskiy et al. , “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929 , 2020
Original
2020
Cited alongside, same era.
L. Wang, Z. Q. Lin, and A. Wong, “Covid-net: A tailored deep convolutional neural network design for detection of covid-19 cases from chest x-ray images,” Scientific reports , vol. 10, no. 1, pp. 1–12, 2020

Then

——, “Laion-5b: An open large-scale dataset for training next generation image-text models,” arXiv preprint arXiv:2210.08402 , 2022
Original
2022
Later among the works it cites.
W. Wang et al. , “Image as a foreign language: Beit pretraining for all vision and vision-language tasks,” arXiv preprint arXiv:2208.10442 , 2022
Original
2022
Later among the works it cites.
H. Xue et al. , “Stare at what you see: Masked image modeling without reconstruction,” arXiv preprint arXiv:2211.08887 , 2022
Original
2022
Later among the works it cites.
N. Mu, A. Kirillov, D. Wagner, and S. Xie, “Slip: Self-supervision meets language-image pre-training,” in European Conference on Computer Vision . Springer, 2022, pp. 529–544
2022
Later among the works it cites.
Y. Gao, J. Liu, Z. Xu, J. Zhang, K. Li, and C. Shen, “Pyramidclip: Hierarchical feature alignment for vision-language model pretraining,” arXiv preprint arXiv:2204.14095 , 2022

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

2020

Cited alongside, same era.

J. Healthcare, “Object-cxr-automatic detection of foreign objects on chest x-rays,” 2020

2020

Cited alongside, same era.

L. Wang, Z. Q. Lin, and A. Wong, “Covid-net: A tailored deep convolutional neural network design for detection of covid-19 cases from chest x-ray images,” Scientific reports , vol. 10, no. 1, pp. 1–12, 2020

2020

Cited alongside, same era.

A. Radford et al. , “Learning transferable visual models from natural language supervision,” in International Conference on Machine Learning . PMLR, 2021, pp. 8748–8763

2021

Cited alongside, same era.

S.-C. Huang, L. Shen, M. P. Lungren, and S. Yeung, “Gloria: A multimodal global-local representation learning framework for label-efficient medical image recognition,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 3942–3951

2021

Cited alongside, same era.

A. Esteva et al. , “Deep learning-enabled medical computer vision,” NPJ digital medicine , vol. 4, no. 1, pp. 1–9, 2021

2021

Cited alongside, same era.

J. Chai, H. Zeng, A. Li, and E. W. Ngai, “Deep learning in computer vision: A critical review of emerging techniques and application scenarios,” Machine Learning with Applications , vol. 6, p. 100134, 2021

2021

Cited alongside, same era.

C. Jia et al. , “Scaling up visual and vision-language representation learning with noisy text supervision,” in International Conference on Machine Learning . PMLR, 2021, pp. 4904–4916

2021

Cited alongside, same era.

Original

2022

Later among the works it cites.

A. Yang, A. Miech, J. Sivic, I. Laptev, and C. Schmid, “Zero-shot video question answering via frozen bidirectional language models,” arXiv preprint arXiv:2206.08155 , 2022

2022

Later among the works it cites.

B. Boecking et al. , “Making the most of text semantics to improve biomedical vision–language processing,” in European conference on computer vision . Springer, 2022, pp. 1–21

2022

Later among the works it cites.

E. Tiu, E. Talius, P. Patel, C. P. Langlotz, A. Y. Ng, and P. Rajpurkar, “Expert-level detection of pathologies from unannotated chest x-ray images via self-supervised learning,” Nature Biomedical Engineering , pp. 1–8, 2022

2022

Later among the works it cites.

A. Saporta et al. , “Benchmarking saliency methods for chest x-ray interpretation,” Nature Machine Intelligence , vol. 4, no. 10, pp. 867–878, 2022

2022

Later among the works it cites.

J. Denize, J. Rabarisoa, A. Orcesi, R. Hérault, and S. Canu, “Similarity contrastive estimation for self-supervised soft contrastive learning,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , 2023, pp. 2706–2716

2023

Closest in time.

R. van der Sluijs, N. Bhaskhar, D. Rubin, C. Langlotz, and A. Chaudhari, “Exploring image augmentations for siamese representation learning with chest x-rays,” arXiv preprint arXiv:2301.12636 , 2023

2023

Closest in time.

J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,” arXiv preprint arXiv:2301.12597 , 2023

2023

Closest in time.

C. Wu, X. Zhang, Y. Zhang, Y. Wang, and W. Xie, “Medklip: Medical knowledge enhanced language-image pre-training,” medRxiv , pp. 2023–01, 2023

2023

Closest in time.

X. Zhang, C. Wu, Y. Zhang, W. Xie, and Y. Wang, “Knowledge-enhanced visual-language pre-training on chest radiology images,” Nature Communications , vol. 14, no. 1, p. 4542, 2023

2023

Closest in time.

Z. Wan et al. , “Med-unic: Unifying cross-lingual medical vision-language pre-training by diminishing bias,” arXiv preprint arXiv:2305.19894 , 2023

Original

2023

Closest in time.

C. Wu, X. Zhang, Y. Zhang, Y. Wang, and W. Xie, “Medklip: Medical knowledge enhanced language-image pre-training,” medRxiv , pp. 2023–01, 2023

2023

Closest in time.

P. Chambon, T. S. Cook, and C. P. Langlotz, “Improved fine-tuning of in-domain transformer model for inferring covid-19 presence in multi-institutional radiology reports,” Journal of Digital Imaging , vol. 36, no. 1, pp. 164–177, 2023

2023

Closest in time.

X. Wang, Y. Peng, L. Lu, Z. Lu, M. Bagheri, and R. M. Summers, “Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2017, pp. 2097–2106

2097

Closest in time.