2023

Segment Anything in Medical Images

Ma, Jun, He, Yuting, Li, Feifei et al.

Understand

Medical image segmentation is a critical component in clinical practice, facilitating accurate diagnosis, treatment planning, and disease monitoring.

However, existing methods, often tailored to specific modalities or disease types, lack generalizability across the diverse spectrum of medical image segmentation tasks.
Here we present MedSAM, a foundation model designed for bridging this gap by enabling universal medical image segmentation.
The model is developed on a large-scale medical image dataset with 1,570,263 image-mask pairs, covering 10 imaging modalities and over 30 cancer types.

Built on

Eisenhauer, E.A., Therasse, P., Bogaerts, J., Schwartz, L.H., Sargent, D., Ford, R., Dancey, J., Arbuck, S., Gwyther, S., Mooney, M., et al
2009
Earlier work this paper cites.
Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., et al
2013
Earlier work this paper cites.
Ba, J.L., Kiros, J.R., Hinton, G.E.: Layer normalization. arXiv preprint arXiv:1607.06450 (2016)
Original
2016
Earlier work this paper cites.
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
2016
Earlier work this paper cites.
Milletari, F., Navab, N., Ahmadi, S.-A.: V-net: Fully convolutional neural networks for volumetric medical image segmentation. In: International Conference on 3D Vision (3DV), pp. 565–571 (2016)
2016
Earlier work this paper cites.
Yushkevich, P.A., Gao, Y., Gerig, G.: Itk-snap: An interactive tool for semi-automatic segmentation of multi-modality biomedical images. In: International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 3342–3345 (2016)
2016
Earlier work this paper cites.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. In: Advances in Neural Information Processing Systems, vol. 30 (2017)
2017
Earlier work this paper cites.
De Fauw, J., Ledsam, J.R., Romera-Paredes, B., Nikolov, S., Tomasev, N., Blackwell, S., Askham, H., Glorot, X., O’Donoghue, B., Visentin, D., et al
2018
Earlier work this paper cites.
Wang, G., Zuluaga, M.A., Li, W., Pratt, R., Patel, P.A., Aertsen, M., Doel, T., David, A.L., Deprest, J., Ourselin, S., et al
2018
Earlier work this paper cites.
Wang, G., Li, W., Zuluaga, M.A., Pratt, R., Patel, P.A., Aertsen, M., Doel, T., David, A.L., Deprest, J., Ourselin, S., et al
2018
Earlier work this paper cites.
Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., Adam, H.: Encoder-decoder with atrous separable convolution for semantic image segmentation. In: Proceedings of the European Conference on Computer Vision, pp. 801–818 (2018)
2018
Earlier work this paper cites.
DeepMind, G.: surface-distance. https://github.com/google-deepmind/surface-distance (2018)
2018
Earlier work this paper cites.
Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: International Conference on Learning Representations (2019)
2019
Earlier work this paper cites.
Iakubovskii, P.: Segmentation Models Pytorch. GitHub (2019)
2019
Earlier work this paper cites.
Ouyang, D., He, B., Ghorbani, A., Yuan, N., Ebinger, J., Langlotz, C.P., Heidenreich, P.A., Harrington, R.A., Liang, D.H., Ashley, E.A., et al
2020
Earlier work this paper cites.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al
2020
Earlier work this paper cites.

Similar

Tancik, M., Srinivasan, P., Mildenhall, B., Fridovich-Keil, S., Raghavan, N., Singhal, U., Ramamoorthi, R., Barron, J., Ng, R.: Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems 33
2020
Cited alongside, same era.
Ahmed, A., Elmohr, M., Fuentes, D., Habra, M., Fisher, S., Perrier, N., Zhang, M., Elsayes, K.: Radiomic mapping model for prediction of ki-67 expression in adrenocortical carcinoma. Clinical Radiology 75
2020
Cited alongside, same era.
Isensee, F., Jaeger, P.F., Kohl, S.A., Petersen, J., Maier-Hein, K.H.: nnu-net: a self-configuring method for deep learning-based biomedical image segmentation. Nature Methods 18
2021
Cited alongside, same era.
Minaee, S., Boykov, Y., Porikli, F., Plaza, A., Kehtarnavaz, N., Terzopoulos, D.: Image segmentation using deep learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence 44
2021
Cited alongside, same era.
Luo, X., Wang, G., Song, T., Zhang, J., Aertsen, M., Deprest, J., Ourselin, S., Vercauteren, T., Zhang, S.: Mideepseg: Minimally interactive segmentation of unseen objects from medical images using deep learning. Medical image analysis 72
2021
Cited alongside, same era.
Ma, J., Chen, J., Ng, M., Huang, R., Li, Y., Li, C., Yang, X., Martel, A.: Loss odyssey in medical image segmentation. Medical Image Analysis 71
2021
Cited alongside, same era.
Antonelli, M., Reinke, A., Bakas, S., Farahani, K., Kopp-Schneider, A., Landman, B.A., Litjens, G., Menze, B., Ronneberger, O., Summers, R.M., et al
2022
Cited alongside, same era.
Bera, K., Braman, N., Gupta, A., Velcheti, V., Madabhushi, A.: Predicting cancer outcomes with radiomics and artificial intelligence in radiology. Nature Reviews Clinical Oncology 19
2022
Cited alongside, same era.
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., Girshick, R.: Masked autoencoders are scalable vision learners. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000–16009 (2022)
2022
Cited alongside, same era.
Maier-Hein, L., Reinke, A., Godau, P., Tizabi, M.D., Büttner, F., et al.: Metrics reloaded: Pitfalls and recommendations for image analysis validation. arXiv preprint arXiv:2206.01653 (2022)
Original
2022
Cited alongside, same era.
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.-Y., Dollar, P., Girshick, R.: Segment anything. In: IEEE International Conference on Computer Vision, pp. 4015–4026 (2023)
2023
Cited alongside, same era.
Zou, X., Yang, J., Zhang, H., Li, F., Li, L., Gao, J., Lee, Y.J.: Segment everything everywhere all at once. In: Advances in Neural Information Processing Systems (2023)
2023
Cited alongside, same era.

Then

He, S., Bao, R., Li, J., Grant, P.E., Ou, Y.: Accuracy of segment-anything model (sam) in medical image segmentation tasks. arXiv preprint arXiv:2304.09324 (2023)
Original
2023
Closest in time.
Wald, T., Roy, S., Koehler, G., Disch, N., Rokuss, M.R., Holzschuh, J., Zimmerer, D., Maier-Hein, K.: SAM.MD: Zero-shot medical image segmentation capabilities of the segment anything model. In: Medical Imaging with Deep Learning, Short Paper Track (2023)
2023
Closest in time.
Zhou, T., Zhang, Y., Zhou, Y., Wu, Y., Gong, C.: Can sam segment polyps? arXiv preprint arXiv:2304.07583 (2023)
Original
2023
Closest in time.
Mohapatra, S., Gosai, A., Schlaug, G.: Sam vs bet: A comparative study for brain extraction and segmentation of magnetic resonance images using deep learning. arXiv preprint arXiv:2304.04738 (2023)
Original
2023
Closest in time.
Chen, J., Bai, X.: Learning to” segment anything” in thermal infrared images through knowledge distillation with a large scale dataset satir. arXiv preprint arXiv:2304.07969 (2023)
Original
2023
Closest in time.
Tang, L., Xiao, H., Li, B.: Can sam segment anything? when sam meets camouflaged object detection. arXiv preprint arXiv:2304.04709 (2023)
Original
2023
Closest in time.
Ji, G.-P., Fan, D.-P., Xu, P., Zhou, B., Cheng, M.-M., Van Gool, L.: Sam struggles in concealed scenes–empirical study on ”segment anything”. Science China Information Sciences 66
2023
Closest in time.
Ji, W., Li, J., Bi, Q., Li, W., Cheng, L.: Segment anything is not always perfect: An investigation of sam on different real-world applications. arXiv preprint arXiv:2304.05750 (2023)
Original
2023
Closest in time.
Mazurowski, M.A., Dong, H., Gu, H., Yang, J., Konz, N., Zhang, Y.: Segment anything model for medical image analysis: an experimental study. Medical Image Analysis 89
2023
Closest in time.
He, B., Kwan, A.C., Cho, J.H., Yuan, N., Pollick, C., Shiota, T., Ebinger, J., Bello, N.A., Wei, J., Josan, K., Duffy, G., Jujjavarapu, M., Siegel, R., Cheng, S., Zou, J.Y., Ouyang, D.: Blinded, randomized trial of sonographer versus AI cardiac function assessment. Nature 616
2023
Closest in time.
Ma, J., Wang, B.: Towards foundation models of biological image segmentation. Nature Methods 20
2023
Closest in time.
Ma, J., Xie, R., Ayyadhury, S., Ge, C., Gupta, A., Gupta, R., Gu, S., Zhang, Y., Lee, G., Kim, J., Lou, W., Li, H., Upschulte, E., Dickscheid, T., Almeida, J.G., Wang, Y., Han, L., Yang, X., Labagnara, M., Rahi, S.J., Kempster, C., Pollitt, A., Espinosa, L., Mignot, T., Middeke, J.M., Eckardt, J.-N., Li, W., Li, Z., Cai, X., Bai, B., Greenwald, N.F., Valen, D.V., Weisbart, E., Cimini, B.A., Li, Z., Zuo, C., Brück, O., Bader, G.D., Wang, B.: The multi-modality cell segmentation challenge: Towards universal solutions. arXiv:2308.05864 (2023)
Original
2023
Closest in time.
Xie, R., Pang, K., Bader, G.D., Wang, B.: Maester: Masked autoencoder guided segmentation at pixel resolution for accurate, self-supervised subcellular structure recognition. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 3292–3301 (2023)
2023
Closest in time.
Moawad, A.W., Ahmed, A.A., ElMohr, M., Eltaher, M., Habra, M.A., Fisher, S., Perrier, N., Zhang, M., Fuentes, D., Elsayes, K.: Voxel-level segmentation of pathologically-proven Adrenocortical carcinoma with Ki-67 expression (Adrenal-ACC-Ki67-Seg) [Data set]. https://doi.org/10.7937/1FPG-VM46 (2023)
2023
Closest in time.
Huang, Y., Yang, X., Liu, L., Zhou, H., Chang, A., Zhou, X., Chen, R., Yu, J., Chen, J., Chen, C., Liu, S., Chi, H., Hu, X., Yue, K., Li, L., Grau, V., Fan, D.-P., Dong, F., Ni, D.: Segment anything model for medical images? Medical Image Analysis 92
2024
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…