2019

On instabilities of deep learning in image reconstruction - Does AI come at a cost?

Antun, Vegard, Renna, Francesco, Poon, Clarice et al.

Understand

Deep learning, due to its unprecedented success in tasks such as image classification, has emerged as a new tool in image reconstruction with potential to change the field.

In this paper we demonstrate a crucial phenomenon: deep learning typically yields unstablemethods for image reconstruction.
The instabilities usually occur in several forms: (1) tiny, almost undetectable perturbations, both in the image and sampling domain, may result in severe artefacts in the reconstruction, (2) a small structural change, for example a tumour, may not be captured in the reconstructed image and (3) (a counterintuitive type of instability) more samples may yield poorer performance.
Our new stability test with algorithms and easy to use software detects the instability phenomena.

Built on

“Sparse MRI: The application of compressed sensing for rapid MR imaging”
Michael Lustig, David Donoho and John˜M Pauly · 1902
Earlier work this paper cites.
“An iteration formula for Fredholm integral equations of the first kind”
Louis Landweber · 1951
Earlier work this paper cites.
“An iteration formula for Fredholm integral equations of the first kind”
Louis Landweber · 1951
Earlier work this paper cites.
“An iteration formula for Fredholm integral equations of the first kind”
Louis Landweber · 1951
Earlier work this paper cites.
“Image reconstruction from incomplete and noisy data”
Stephen˜F Gull and Geoff˜J Daniell · 1978
Earlier work this paper cites.
“Image reconstruction from incomplete and noisy data”
Stephen˜F Gull and Geoff˜J Daniell · 1978
Earlier work this paper cites.
“Image reconstruction from incomplete and noisy data”
Stephen˜F Gull and Geoff˜J Daniell · 1978
Earlier work this paper cites.
“Nonlinear total variation based noise removal algorithms”
Leonid˜I Rudin, Stanley Osher and Emad Fatemi · 1992
Earlier work this paper cites.
“Nonlinear total variation based noise removal algorithms”
Leonid˜I Rudin, Stanley Osher and Emad Fatemi · 1992
Earlier work this paper cites.
“Nonlinear total variation based noise removal algorithms”
Leonid˜I Rudin, Stanley Osher and Emad Fatemi · 1992
Earlier work this paper cites.
“Regularization of Inverse Problems”, Mathematics and Its Applications
H.W. Engl, M. Hanke and A. Neubauer · 1996
Earlier work this paper cites.
“Regularization of Inverse Problems”, Mathematics and Its Applications
H.W. Engl, M. Hanke and A. Neubauer · 1996
Earlier work this paper cites.
“Regularization of Inverse Problems”, Mathematics and Its Applications
H.W. Engl, M. Hanke and A. Neubauer · 1996
Earlier work this paper cites.
“SENSE: sensitivity encoding for fast MRI”
Klaas˜P Pruessmann, Markus Weiger, Markus˜B Scheidegger and Peter Boesiger · 1999
Earlier work this paper cites.
“SENSE: sensitivity encoding for fast MRI”
Klaas˜P Pruessmann, Markus Weiger, Markus˜B Scheidegger and Peter Boesiger · 1999
Earlier work this paper cites.
“SENSE: sensitivity encoding for fast MRI”
Klaas˜P Pruessmann, Markus Weiger, Markus˜B Scheidegger and Peter Boesiger · 1999
Earlier work this paper cites.
“The L-Curve and its Use in the Numerical Treatment of Inverse Problems”
P.˜C. Hansen · 2000
Earlier work this paper cites.
“The L-Curve and its Use in the Numerical Treatment of Inverse Problems”
P.˜C. Hansen · 2000
Earlier work this paper cites.
“The L-Curve and its Use in the Numerical Treatment of Inverse Problems”
P.˜C. Hansen · 2000
Earlier work this paper cites.
“Best practices for convolutional neural networks applied to visual document analysis”
P.˜Y. Simard, D. Steinkraus and J.˜C. Platt · 2003
Earlier work this paper cites.
“Best practices for convolutional neural networks applied to visual document analysis”
P.˜Y. Simard, D. Steinkraus and J.˜C. Platt · 2003
Earlier work this paper cites.
“Best practices for convolutional neural networks applied to visual document analysis”
P.˜Y. Simard, D. Steinkraus and J.˜C. Platt · 2003
Earlier work this paper cites.
“Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information”
E.˜J. Cand\‘es, J. Romberg and T. Tao · 2006
Earlier work this paper cites.
“Compressed sensing”
D.˜L. Donoho · 2006
Earlier work this paper cites.
“Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information”
E.˜J. Cand\‘es, J. Romberg and T. Tao · 2006
Earlier work this paper cites.
“Compressed sensing”
D.˜L. Donoho · 2006
Earlier work this paper cites.
“Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information”
E.˜J. Cand\‘es, J. Romberg and T. Tao · 2006
Earlier work this paper cites.
“Compressed sensing”
D.˜L. Donoho · 2006
Earlier work this paper cites.
“Improved k–t BLAST and k–t SENSE using FOCUSS”
Hong Jung, Jong˜Chul Ye and Eung˜Yeop Kim · 2007
Earlier work this paper cites.
“Sparse MRI: The application of compressed sensing for rapid MR imaging”
Michael Lustig, David Donoho and John˜M Pauly · 2007
Earlier work this paper cites.
“Improved k–t BLAST and k–t SENSE using FOCUSS”
Hong Jung, Jong˜Chul Ye and Eung˜Yeop Kim · 2007
Earlier work this paper cites.
“Sparse MRI: The application of compressed sensing for rapid MR imaging”
Michael Lustig, David Donoho and John˜M Pauly · 2007
Earlier work this paper cites.
“Improved k–t BLAST and k–t SENSE using FOCUSS”
Hong Jung, Jong˜Chul Ye and Eung˜Yeop Kim · 2007
Earlier work this paper cites.
“Probing the Pareto frontier for basis pursuit solutions”
Ewout Berg and Michael˜P Friedlander · 2008
Earlier work this paper cites.
“Scope of validity of PSNR in image/video quality assessment”
Quan Huynh-Thu and Mohammed Ghanbari · 2008
Earlier work this paper cites.
“Probing the Pareto frontier for basis pursuit solutions”
Ewout Berg and Michael˜P Friedlander · 2008
Earlier work this paper cites.
“Scope of validity of PSNR in image/video quality assessment”
Quan Huynh-Thu and Mohammed Ghanbari · 2008
Earlier work this paper cites.
“Probing the Pareto frontier for basis pursuit solutions”
Ewout Berg and Michael˜P Friedlander · 2008
Earlier work this paper cites.
“Scope of validity of PSNR in image/video quality assessment”
Quan Huynh-Thu and Mohammed Ghanbari · 2008
Earlier work this paper cites.
“Fields of experts”
Stefan Roth and Michael˜J Black · 2009
Earlier work this paper cites.
“Fields of experts”
Stefan Roth and Michael˜J Black · 2009
Earlier work this paper cites.
“Fields of experts”
Stefan Roth and Michael˜J Black · 2009
Earlier work this paper cites.
“Compressive Fluorescence Microscopy for Biological and Hyperspectral Imaging”
V. Studer et al · 2011
Earlier work this paper cites.
“Compressive Fluorescence Microscopy for Biological and Hyperspectral Imaging”
V. Studer et al · 2011
Earlier work this paper cites.
“Compressive Fluorescence Microscopy for Biological and Hyperspectral Imaging”
V. Studer et al · 2011
Earlier work this paper cites.
“Imagenet classification with deep convolutional neural networks”
Alex Krizhevsky, Ilya Sutskever and Geoffrey˜E Hinton · 2012
Earlier work this paper cites.
“Image denoising: Can plain neural networks compete with BM3D?”
H.˜C. Burger, C.˜J. Schuler and S. Harmeling · 2012
Earlier work this paper cites.

Similar

“Imagenet classification with deep convolutional neural networks”
Alex Krizhevsky, Ilya Sutskever and Geoffrey˜E Hinton · 2012
Cited alongside, same era.
“Image denoising: Can plain neural networks compete with BM3D?”
H.˜C. Burger, C.˜J. Schuler and S. Harmeling · 2012
Cited alongside, same era.
“Imagenet classification with deep convolutional neural networks”
Alex Krizhevsky, Ilya Sutskever and Geoffrey˜E Hinton · 2012
Cited alongside, same era.
“Image denoising: Can plain neural networks compete with BM3D?”
H.˜C. Burger, C.˜J. Schuler and S. Harmeling · 2012
Cited alongside, same era.
“Rich feature hierarchies for accurate object detection and semantic segmentation”
Ross Girshick, Jeff Donahue, Trevor Darrell and Jitendra Malik · 2014
Cited alongside, same era.
“Learning deep features for scene recognition using places database”
Bolei Zhou et al · 2014
Cited alongside, same era.
“Intriguing properties of neural networks”
Christian Szegedy et al · 2014
Cited alongside, same era.
“Dictionary learning and time sparsity for dynamic MR data reconstruction”
Jose Caballero, Anthony˜N Price, Daniel Rueckert and Joseph˜V Hajnal · 2014
Cited alongside, same era.
“ESPIRiT-an eigenvalue approach to autocalibrating parallel MRI: where SENSE meets GRAPPA”
Martin Uecker et al · 2014
Cited alongside, same era.
“Rich feature hierarchies for accurate object detection and semantic segmentation”
Ross Girshick, Jeff Donahue, Trevor Darrell and Jitendra Malik · 2014
Cited alongside, same era.
“Learning deep features for scene recognition using places database”
Bolei Zhou et al · 2014
Cited alongside, same era.
“Intriguing properties of neural networks”
Christian Szegedy et al · 2014
Cited alongside, same era.

Then

“Deep convolutional neural network for inverse problems in imaging”
Kyong˜Hwan Jin, Michael˜T McCann, Emmanuel Froustey and Michael Unser · 2017
Later among the works it cites.
“A deep cascade of convolutional neural networks for MR image reconstruction”
Jo Schlemper et al · 2017
Later among the works it cites.
“DAGAN: Deep De-Aliasing Generative Adversarial Networks for Fast Compressed Sensing MRI Reconstruction”
Guang Yang et al · 2017
Later among the works it cites.
“Deep, Deep Trouble - Deep Learning’s Impact on Image Processing, Mathematics, and Humanity”
M. Elad · 2017
Later among the works it cites.
“Universal Adversarial Perturbations”
S. Moosavi-Dezfooli, A. Fawzi, O. Fawzi and P. Frossard · 2017
Later among the works it cites.
“The Robustness of Deep Networks – A geometric perspective”
Alhussein Fawzi, Seyed˜Mohsen Moosavi˜Dezfooli and Pascal Frossard · 2017
Later among the works it cites.
“Breaking the coherence barrier: A new theory for compressed sensing.”
Ben Adcock, Anders˜C. Hansen, Clarice Poon and Bogdan Roman · 2017
Later among the works it cites.
“Variational networks: connecting variational methods and deep learning”
Erich Kobler, Teresa Klatzer, Kerstin Hammernik and Thomas Pock · 2017
Later among the works it cites.
“Convolutional Neural Networks for Inverse Problems in Imaging: A Review”
M.˜T. McCann, K.˜H. Jin and M. Unser · 2017
Later among the works it cites.
“Deep convolutional neural network for inverse problems in imaging”
Kyong˜Hwan Jin, Michael˜T McCann, Emmanuel Froustey and Michael Unser · 2017
Later among the works it cites.
“A deep cascade of convolutional neural networks for MR image reconstruction”
Jo Schlemper et al · 2017
Later among the works it cites.
“DAGAN: Deep De-Aliasing Generative Adversarial Networks for Fast Compressed Sensing MRI Reconstruction”
Guang Yang et al · 2017
Later among the works it cites.
“Deep, Deep Trouble - Deep Learning’s Impact on Image Processing, Mathematics, and Humanity”
M. Elad · 2017
Later among the works it cites.
“Universal Adversarial Perturbations”
S. Moosavi-Dezfooli, A. Fawzi, O. Fawzi and P. Frossard · 2017
Later among the works it cites.
“The Robustness of Deep Networks – A geometric perspective”
Alhussein Fawzi, Seyed˜Mohsen Moosavi˜Dezfooli and Pascal Frossard · 2017
Later among the works it cites.
“Breaking the coherence barrier: A new theory for compressed sensing.”
Ben Adcock, Anders˜C. Hansen, Clarice Poon and Bogdan Roman · 2017
Later among the works it cites.
“Variational networks: connecting variational methods and deep learning”
Erich Kobler, Teresa Klatzer, Kerstin Hammernik and Thomas Pock · 2017
Later among the works it cites.
“Convolutional Neural Networks for Inverse Problems in Imaging: A Review”
M.˜T. McCann, K.˜H. Jin and M. Unser · 2017
Later among the works it cites.
“Deep convolutional neural network for inverse problems in imaging”
Kyong˜Hwan Jin, Michael˜T McCann, Emmanuel Froustey and Michael Unser · 2017
Later among the works it cites.
“A deep cascade of convolutional neural networks for MR image reconstruction”
Jo Schlemper et al · 2017
Later among the works it cites.
“DAGAN: Deep De-Aliasing Generative Adversarial Networks for Fast Compressed Sensing MRI Reconstruction”
Guang Yang et al · 2017
Later among the works it cites.
“Deep, Deep Trouble - Deep Learning’s Impact on Image Processing, Mathematics, and Humanity”
M. Elad · 2017
Later among the works it cites.
“Universal Adversarial Perturbations”
S. Moosavi-Dezfooli, A. Fawzi, O. Fawzi and P. Frossard · 2017
Later among the works it cites.
“The Robustness of Deep Networks – A geometric perspective”
Alhussein Fawzi, Seyed˜Mohsen Moosavi˜Dezfooli and Pascal Frossard · 2017
Later among the works it cites.
“Breaking the coherence barrier: A new theory for compressed sensing.”
Ben Adcock, Anders˜C. Hansen, Clarice Poon and Bogdan Roman · 2017
Later among the works it cites.
“Variational networks: connecting variational methods and deep learning”
Erich Kobler, Teresa Klatzer, Kerstin Hammernik and Thomas Pock · 2017
Later among the works it cites.
“Image reconstruction by domain-transform manifold learning”
Bo Zhu et al · 2018
Later among the works it cites.
“AI transforms image reconstruction”
Rita Strack · 2018
Later among the works it cites.
“Learning a variational network for reconstruction of accelerated MRI data”
Kerstin Hammernik et al · 2018
Later among the works it cites.
“Using deep neural networks for inverse problems in imaging: beyond analytical methods”
Alice Lucas, Michael Iliadis, Rafael Molina and Aggelos˜K Katsaggelos · 2018
Later among the works it cites.
“Neural Proximal Gradient Descent for Compressive Imaging”
Morteza Mardani et al · 2018
Later among the works it cites.
“Geometric robustness of deep networks: analysis and improvement”
Can Kanbak, Seyed-Mohsen Moosavi-Dezfooli and Pascal Frossard · 2018
Later among the works it cites.
“Shearlet-based compressed sensing for fast 3D cardiac MR imaging using iterative reweighting”
Jackie Ma et al · 2018
Later among the works it cites.
“Image reconstruction by domain-transform manifold learning”
Bo Zhu et al · 2018
Later among the works it cites.
“AI transforms image reconstruction”
Rita Strack · 2018
Later among the works it cites.
“Learning a variational network for reconstruction of accelerated MRI data”
Kerstin Hammernik et al · 2018
Later among the works it cites.
“Using deep neural networks for inverse problems in imaging: beyond analytical methods”
Alice Lucas, Michael Iliadis, Rafael Molina and Aggelos˜K Katsaggelos · 2018
Later among the works it cites.
“Neural Proximal Gradient Descent for Compressive Imaging”
Morteza Mardani et al · 2018
Later among the works it cites.
“Geometric robustness of deep networks: analysis and improvement”
Can Kanbak, Seyed-Mohsen Moosavi-Dezfooli and Pascal Frossard · 2018
Later among the works it cites.
“Shearlet-based compressed sensing for fast 3D cardiac MR imaging using iterative reweighting”
Jackie Ma et al · 2018
Later among the works it cites.
“Image reconstruction by domain-transform manifold learning”
Bo Zhu et al · 2018
Later among the works it cites.
“AI transforms image reconstruction”
Rita Strack · 2018
Later among the works it cites.
“Learning a variational network for reconstruction of accelerated MRI data”
Kerstin Hammernik et al · 2018
Later among the works it cites.
“Using deep neural networks for inverse problems in imaging: beyond analytical methods”
Alice Lucas, Michael Iliadis, Rafael Molina and Aggelos˜K Katsaggelos · 2018
Later among the works it cites.
“Neural Proximal Gradient Descent for Compressive Imaging”
Morteza Mardani et al · 2018
Later among the works it cites.
“Geometric robustness of deep networks: analysis and improvement”
Can Kanbak, Seyed-Mohsen Moosavi-Dezfooli and Pascal Frossard · 2018
Later among the works it cites.
“Shearlet-based compressed sensing for fast 3D cardiac MR imaging using iterative reweighting”
Jackie Ma et al · 2018
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…