2022

Unsupervised Medical Image Translation with Adversarial Diffusion Models

Özbey, Muzaffer, Dalmaz, Onat, Dar, Salman UH et al.

Understand

Imputation of missing images via source-to-target modality translation can improve diversity in medical imaging protocols.

A pervasive approach for synthesizing target images involves one-shot mapping through generative adversarial networks (GAN).
Yet, GAN models that implicitly characterize the image distribution can suffer from limited sample fidelity.
Here, we propose a novel method based on adversarial diffusion modeling, SynDiff, for improved performance in medical image translation.

Built on

W. Feller, “On the theory of stochastic processes, with particular reference to applications,” in Proc Berkeley Symp Math Stat Probab , vol. 1. University of California Press, 1949, pp. 403–433
1949
Earlier work this paper cites.
M. Jenkinson and S. Smith, “A global optimisation method for robust affine registration of brain images,” Med Image Anal , vol. 5, pp. 143–156, 2001
2001
Earlier work this paper cites.
J. E. Iglesias, E. Konukoglu, D. Zikic, B. Glocker, K. Van Leemput, and B. Fischl, “Is synthesizing MRI contrast useful for inter-modality analysis?” in Med Image Comput Comput Assist Interv , 2013, pp. 631–638
2013
Earlier work this paper cites.
D. H. Ye, D. Zikic, B. Glocker, A. Criminisi, and E. Konukoglu, “Modality propagation: Coherent synthesis of subject-specific scans with data-driven regularization,” in Med Image Comput Comput Assist Interv , 2013, pp. 606–613
2013
Earlier work this paper cites.
S. Roy, A. Jog, A. Carass, and J. L. Prince, “Atlas based intensity transformation of brain MR images,” in Multimodal Brain Image Anal. , 2013, pp. 51–62
2013
Earlier work this paper cites.
D. C. Alexander, D. Zikic, J. Zhang, H. Zhang, and A. Criminisi, “Image quality transfer via random forest regression: Applications in diffusion MRI,” in Med Image Comput Comput Assist Interv , 2014, pp. 225–232
2014
Earlier work this paper cites.
I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair et al. , “Generative adversarial nets,” Adv Neural Inf Process Syst , vol. 27, 2014
2014
Earlier work this paper cites.
H. Van Nguyen, K. Zhou, and R. Vemulapalli, “Cross-domain synthesis of medical images using efficient location-sensitive deep network,” in Med Image Comput Comput Assist Interv , 2015, pp. 677–684
2015
Earlier work this paper cites.
R. Vemulapalli, H. Van Nguyen, and S. K. Zhou, “Unsupervised cross-modal synthesis of subject-specific scans,” in Int Conf Comput Vis , 2015, pp. 630–638
2015
Earlier work this paper cites.
J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli, “Deep unsupervised learning using nonequilibrium thermodynamics,” in Int Conf Mach Learn , 2015, pp. 2256–2265
2015
Earlier work this paper cites.
B. H. Menze, A. Jakab, S. Bauer, J. Kalpathy-Cramer, K. Farahani, J. Kirby et al. , “The multimodal brain tumor image segmentation benchmark (BRATS),” IEEE Trans Med Imag , vol. 34, no. 10, pp. 1993–2024, 2015
2015
Earlier work this paper cites.
O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in Med Image Comput Comput Assist Inter . Springer, 2015, pp. 234–241
2015
Earlier work this paper cites.
T. Huynh, Y. Gao, J. Kang, L. Wang, P. Zhang, J. Lian et al. , “Estimating CT image from MRI data using structured random forest and auto-context model,” IEEE Trans Med Imag , vol. 35, no. 1, pp. 174–183, 2016
2016
Earlier work this paper cites.
N. Cordier, H. Delingette, M. Le, and N. Ayache, “Extended modality propagation: Image synthesis of pathological cases,” IEEE Trans Med Imag , vol. 35, pp. 2598–2608, 2016
2016
Earlier work this paper cites.
Y. Wu, W. Yang, L. Lu, Z. Lu, L. Zhong, M. Huang et al. , “Prediction of CT substitutes from MR images based on local diffeomorphic mapping for brain PET attenuation correction,” J Nucl Med , vol. 57, no. 10, pp. 1635–1641, 2016
2016
Earlier work this paper cites.
V. Sevetlidis, M. V. Giuffrida, and S. A. Tsaftaris, “Whole image synthesis using a deep encoder-decoder network,” in Simul Synth Med Imaging , 2016, pp. 127–137
2016
Earlier work this paper cites.
D. Nie, X. Cao, Y. Gao, L. Wang, and D. Shen, “Estimating CT image from MRI data using 3D fully convolutional networks,” in Deep Learn Data Label Med Appl , 2016, pp. 170–178
2016
Earlier work this paper cites.
C. Bowles, C. Qin, C. Ledig, R. Guerrero, R. Gunn, A. Hammers et al. , “Pseudo-healthy image synthesis for white matter lesion segmentation,” in Simul Synth Med Imaging , 2016, pp. 87–96
2016
Earlier work this paper cites.
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in IEEE Conf Comput Vis Pattern Recognit , 2016, pp. 770–778
2016
Earlier work this paper cites.
J. Lee, A. Carass, A. Jog, C. Zhao, and J. Prince, “Multi-atlas-based CT synthesis from conventional MRI with patch-based refinement for MRI-based radiotherapy planning,” in SPIE Med Imag. , vol. 10133, 2017, p. 101331I
2017
Earlier work this paper cites.
A. Jog, A. Carass, S. Roy, D. L. Pham, and J. L. Prince, “Random forest regression for magnetic resonance image synthesis,” Med Image Anal , vol. 35, pp. 475–488, 2017
2017
Earlier work this paper cites.
T. Joyce, A. Chartsias, and S. A. Tsaftaris, “Robust multi-modal MR image synthesis,” in Med Image Comput Comput Assist Interv , 2017, pp. 347–355
2017
Earlier work this paper cites.
C. Zhao, A. Carass, J. Lee, Y. He, and J. L. Prince, “Whole brain segmentation and labeling from CT using synthetic MR images,” in Mach Learn Med Imaging , 2017, pp. 291–298
2017
Earlier work this paper cites.
Y. Huang, L. Shao, and A. F. Frangi, “Simultaneous super-resolution and cross-modality synthesis of 3D medical images using weakly-supervised joint convolutional sparse coding,” Comput Vis Pattern Recognit , pp. 5787–5796, 2017
2017
Earlier work this paper cites.
P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, “Image-to-image translation with conditional adversarial networks,” Comput Vis Pattern Recognit , pp. 1125–1134, 2017
2017
Earlier work this paper cites.
A. Chartsias, T. Joyce, R. Dharmakumar, and S. A. Tsaftaris, “Adversarial image synthesis for unpaired multi-modal cardiac data,” in Simul Synth Med Imaging , 2017, pp. 3–13
2017
Earlier work this paper cites.
J. M. Wolterink, A. M. Dinkla, M. H. F. Savenije, P. R. Seevinck, C. A. T. van den Berg, and I. Išgum, “Deep MR to CT synthesis using unpaired data,” in Simul Synth Med Imaging , Cham, 2017, pp. 14–23
2017
Earlier work this paper cites.
M.-Y. Liu, T. Breuel, and J. Kautz, “Unsupervised image-to-image translation networks,” in Adv Neural Inf Process Syst , vol. 30, 2017
2017
Earlier work this paper cites.

Similar

Y. Huang, L. Shao, and A. F. Frangi, “Cross-modality image synthesis via weakly coupled and geometry co-regularized joint dictionary learning,” IEEE Trans Med Imag , vol. 37, no. 3, pp. 815–827, 2018
2018
Cited alongside, same era.
A. Chartsias, T. Joyce, M. V. Giuffrida, and S. A. Tsaftaris, “Multimodal MR synthesis via modality-invariant latent representation,” IEEE Trans Med Imag , vol. 37, no. 3, pp. 803–814, 2018
2018
Cited alongside, same era.
B. Yu, L. Zhou, L. Wang, J. Fripp, and P. Bourgeat, “3D cGAN based cross-modality MR image synthesis for brain tumor segmentation,” Int. Symp. Biomed. Imaging , pp. 626–630, 2018
2018
Cited alongside, same era.
D. Nie, R. Trullo, J. Lian, L. Wang, C. Petitjean, S. Ruan et al. , “Medical image synthesis with deep convolutional adversarial networks,” IEEE Trans. Biomed. Eng. , vol. 65, no. 12, pp. 2720–2730, 2018
2018
Cited alongside, same era.
H. Yang, J. Sun, A. Carass, C. Zhao, J. Lee, Z. Xu et al. , “Unpaired brain MR-to-CT synthesis using a structure-constrained cycleGAN,” arXiv:1809.04536 , 2018
Original
2018
Cited alongside, same era.
Y. Hiasa, Y. Otake, M. Takao, T. Matsuoka, K. Takashima, A. Carass et al. , “Cross-modality image synthesis from unpaired data using cycleGAN: Effects of gradient consistency loss and training data size,” in Simul Synth Med Imaging , 2018, pp. 31–41
2018
Cited alongside, same era.
L. Mescheder, A. Geiger, and S. Nowozin, “Which training methods for gans do actually converge?” in Int Conf Mach Learn , 2018, pp. 3481–3490
2018
Cited alongside, same era.
T. Nyholm, S. Svensson, S. Andersson, J. Jonsson, M. Sohlin, C. Gustafsson et al. , “MR and CT data with multiobserver delineations of organs in the pelvic area—part of the gold atlas project,” Med Phys , vol. 45, no. 3, pp. 1295–1300, 2018
2018
Cited alongside, same era.
K. H. Kim, W.-J. Do, and S.-H. Park, “Improving resolution of mr images with an adversarial network incorporating images with different contrast,” Med Phys , vol. 45, no. 7, pp. 3120–3131, 2018
2018
Cited alongside, same era.
X. Huang, M.-Y. Liu, S. Belongie, and J. Kautz, “Multimodal unsupervised image-to-image translation,” in European Conf Comput Vis , 2018, pp. 172–189
2018
Cited alongside, same era.
O. Oktay, J. Schlemper, L. L. Folgoc, M. J. Lee, M. Heinrich, K. Misawa et al. , “Attention U-Net: Learning where to look for the pancreas,” arXiv:1804.03999 , 2018
Original
2018
Cited alongside, same era.
Q. Yang, P. Yan, Y. Zhang, H. Yu, Y. Shi, X. Mou et al. , “Low-dose ct image denoising using a generative adversarial network with wasserstein distance and perceptual loss,” IEEE Trans Med Imag , vol. 37, no. 6, pp. 1348–1357, 2018
2018
Cited alongside, same era.

Then

J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” in Adv Neural Inf Process Syst , vol. 33, 2020, pp. 6840–6851
2020
Later among the works it cites.
J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” Adv Neural Inf Process Syst , vol. 33, pp. 6840–6851, 2020
2020
Later among the works it cites.
Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole, “Score-based generative modeling through stochastic differential equations,” arXiv:2011.13456 , 2020
Original
2020
Later among the works it cites.
M. Yurt, S. U. Dar, A. Erdem, E. Erdem, K. K. Oguz, and T. Çukur, “mustGAN: multi-stream generative adversarial networks for MR image synthesis,” Med Image Anal , vol. 70, p. 101944, 2021
2021
Later among the works it cites.
P. Dhariwal and A. Nichol, “Diffusion models beat gans on image synthesis,” in Adv Neural Inf Process Syst , vol. 34, 2021, pp. 8780–8794
2021
Later among the works it cites.
A. Jalal, M. Arvinte, G. Daras, E. Price, A. G. Dimakis, and J. Tamir, “Robust compressed sensing mri with deep generative priors,” in Adv Neural Inf Process Syst , vol. 34, 2021, pp. 14 938–14 954
2021
Later among the works it cites.
H. Sasaki, C. G. Willcocks, and T. P. Breckon, “Unit-DDPM: Unpaired image translation with denoising diffusion probabilistic models,” arXiv:2104.05358 , 2021
Original
2021
Later among the works it cites.
A. Q. Nichol and P. Dhariwal, “Improved denoising diffusion probabilistic models,” in Int Conf Mach Learn , 2021, pp. 8162–8171
2021
Later among the works it cites.
Y. Luo, Y. Wang, C. Zu, B. Zhan, X. Wu, J. Zhou et al. , “3D Transformer-GAN for high-quality PET reconstruction,” in Med Image Comput Comput Assist Interv , 2021, pp. 276–285
2021
Later among the works it cites.
F. Wu and X. Zhuang, “Unsupervised domain adaptation with variational approximation for cardiac segmentation,” IEEE Trans Med Imag , vol. 40, no. 12, pp. 3555–3567, 2021
2021
Later among the works it cites.
O. Dalmaz, M. Yurt, and T. Çukur, “ResViT: Residual vision transformers for multi-modal medical image synthesis,” IEEE Trans Med Imaging , vol. 44, no. 10, pp. 2598–2614, 2022
2022
Closest in time.
H. Chung and J. C. Ye, “Score-based diffusion models for accelerated mri,” Med Image Anal , vol. 80, p. 102479, 2022
2022
Closest in time.
Y. Song, L. Shen, L. Xing, and S. Ermon, “Solving inverse problems in medical imaging with score-based generative models,” in Int Conf Learn Represent , 2022
2022
Closest in time.
S. U. Dar, Şaban Öztürk, Y. Korkmaz, G. Elmas, M. Özbey, A. Güngör et al. , “Adaptive diffusion priors for accelerated mri reconstruction,” arXiv:2207.05876 , 2022
Original
2022
Closest in time.
H. Chung, E. S. Lee, and J. C. Ye, “Mr image denoising and super-resolution using regularized reverse diffusion,” arXiv:2203.12621 , 2022
Original
2022
Closest in time.
W. H. L. Pinaya, P.-D. Tudosiu, J. Dafflon, P. F. da Costa, V. Fernandez, P. Nachev et al. , “Brain imaging generation with latent diffusion models,” arXiv:2209.07162 , 2022
Original
2022
Closest in time.
J. Wolleb, F. Bieder, R. Sandkühler, and P. C. Cattin, “Diffusion models for medical anomaly detection,” in Med Image Comput Comput Assist Inter , vol. 13438. Springer, 2022, pp. 35–45
2022
Closest in time.
W. H. L. Pinaya, M. S. Graham, R. Gray, P. F. Da Costa, P.-D. Tudosiu, P. Wright et al. , “Fast Unsupervised Brain Anomaly Detection and Segmentation with Diffusion Models,” arXiv:2206.03461 , 2022
Original
2022
Closest in time.
Z. Xiao, K. Kreis, and A. Vahdat, “Tackling the generative learning trilemma with denoising diffusion GANs,” in Int Conf Learn Represent , 2022
2022
Closest in time.
X. Meng, Y. Gu, Y. Pan, N. Wang, P. Xue, M. Lu et al. , “A novel unified conditional score-based generative framework for multi-modal medical image completion,” arXiv:2207.03430 , 2022
Original
2022
Closest in time.
M. Yurt, O. Dalmaz, S. U. H. Dar, M. Özbey, B. Tınaz, K. K. Oğuz et al. , “Semi-supervised learning of MRI synthesis without fully-sampled ground truths,” IEEE Trans Med Imaging , vol. 41, no. 12, pp. 3895–3906, 2022
2022
Closest in time.
H. Chung, B. Sim, and J. C. Ye, “Come-closer-diffuse-faster: Accelerating conditional diffusion models for inverse problems through stochastic contraction,” in IEEE Conf Comput Vis Pattern Recognit , 2022, pp. 12 413–12 422
2022
Closest in time.
Q. Lyu and G. Wang, “Conversion between CT and MRI images using diffusion and score-matching models,” arXiv:2209.12104 , 2022
Original
2022
Closest in time.
Y. Korkmaz, S. U. H. Dar, M. Yurt, M. Ozbey, and T. Cukur, “Unsupervised MRI reconstruction via zero-shot learned adversarial transformers,” IEEE Trans Med Imaging , vol. 41, no. 7, pp. 1747–1763, 2022
2022
Closest in time.
A. Gungor, B. Askin, D. A. Soydan, E. U. Saritas, C. B. Top, and T. Çukur, “TranSMS: Transformers for super-resolution calibration in magnetic particle imaging,” IEEE Trans Med Imaging , vol. 41, no. 12, pp. 3562–3574, 2022
2022
Closest in time.
F. Wu, L. Li, and X. Zhuang, “Multi-modality cardiac segmentation via mixing domains for unsupervised adaptation,” in STACOM: Int Work Stat Atl Comp Mod Heart , 2022, pp. 179–188
2022
Closest in time.
J. Gao, J. Zhang, X. Liu, T. Darrell, E. Shelhamer, and D. Wang, “Back to the source: Diffusion-driven test-time adaptation,” arXiv:2207.03442 , 2022
Original
2022
Closest in time.
T. Xiang, M. Yurt, A. B. Syed, K. Setsompop, and A. Chaudhari, “DDM 2 : Self-Supervised Diffusion MRI Denoising with Generative Diffusion Models,” arXiv:2302.03018 , 2023
Original
2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…