2017

The Perception-Distortion Tradeoff

Blau, Yochai, Michaeli, Tomer

Understand

Image restoration algorithms are typically evaluated by some distortion measure (e.g.

PSNR, SSIM, IFC, VIF) or by human opinion scores that quantify perceived perceptual quality.
In this paper, we prove mathematically that distortion and perceptual quality are at odds with each other.
Specifically, we study the optimal probability for correctly discriminating the outputs of an image restoration algorithm from real images.

Built on

Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” Proceedings of the IEEE , vol. 86, no. 11, pp. 2278–2324, 1998
1998
Earlier work this paper cites.
D. Martin, C. Fowlkes, D. Tal, and J. Malik, “A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics,” in International Conference on Computer Vision (ICCV) , vol. 2, 2001, pp. 416–423
2001
Earlier work this paper cites.
S. Baker and T. Kanade, “Limits on super-resolution and how to break them,” IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) , vol. 24, no. 9, pp. 1167–1183, 2002
2002
Earlier work this paper cites.
Z. Wang, E. P. Simoncelli, and A. C. Bovik, “Multiscale structural similarity for image quality assessment,” in Conference on Signals, Systems and Computers , vol. 2, 2003, pp. 1398–1402
2003
Earlier work this paper cites.
Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: From error visibility to structural similarity,” IEEE Transactions on Image Processing , vol. 13, no. 4, pp. 600–612, 2004
2004
Earlier work this paper cites.
I. Csiszár, P. C. Shields et al. , “Information theory and statistics: A tutorial,” Foundations and Trends® in Communications and Information Theory , vol. 1, no. 4, pp. 417–528, 2004
2004
Earlier work this paper cites.
H. R. Sheikh, A. C. Bovik, and G. De Veciana, “An information fidelity criterion for image quality assessment using natural scene statistics,” IEEE Transactions on Image Processing , vol. 14, no. 12, pp. 2117–2128, 2005
2005
Earlier work this paper cites.
Z. Wang and E. P. Simoncelli, “Reduced-reference image quality assessment using a wavelet-domain natural image statistic model.” in Human Vision and Electronic Imaging , vol. 5666, 2005, pp. 149–159
2005
Earlier work this paper cites.
H. R. Sheikh and A. C. Bovik, “Image information and visual quality,” IEEE Transactions on Image Processing , vol. 15, no. 2, pp. 430–444, 2006
2006
Earlier work this paper cites.
Z. Wang, G. Wu, H. R. Sheikh, E. P. Simoncelli, E.-H. Yang, and A. C. Bovik, “Quality-aware images,” IEEE Transactions on Image Processing , vol. 15, no. 6, pp. 1680–1689, 2006
2006
Earlier work this paper cites.
D. M. Chandler and S. S. Hemami, “VSNR: A wavelet-based visual signal-to-noise ratio for natural images,” IEEE Transactions on Image Processing , vol. 16, no. 9, pp. 2284–2298, 2007
2007
Earlier work this paper cites.
M. Nikolova, “Model distortions in Bayesian MAP reconstruction,” Inverse Problems and Imaging , vol. 1, no. 2, p. 399, 2007
2007
Earlier work this paper cites.
Z. Wang and A. C. Bovik, “Mean squared error: Love it or leave it? A new look at signal fidelity measures,” IEEE Signal Processing Magazine , vol. 26, no. 1, pp. 98–117, 2009
2009
Earlier work this paper cites.
Q. Li and Z. Wang, “Reduced-reference image quality assessment using divisive normalization-based image representation,” IEEE Journal of Selected Topics in Signal Processing , vol. 3, no. 2, pp. 202–211, 2009
2009
Earlier work this paper cites.
P. Chatterjee and P. Milanfar, “Is denoising dead?” IEEE Transactions on Image Processing , vol. 19, no. 4, pp. 895–911, 2010
2010
Earlier work this paper cites.
L. Zhang, L. Zhang, X. Mou, and D. Zhang, “FSIM: A feature similarity index for image quality assessment,” IEEE Transactions on Image Processing , vol. 20, no. 8, pp. 2378–2386, 2011
2011
Earlier work this paper cites.
A. K. Moorthy and A. C. Bovik, “Blind image quality assessment: From natural scene statistics to perceptual quality,” IEEE transactions on Image Processing , vol. 20, no. 12, pp. 3350–3364, 2011
2011
Earlier work this paper cites.
A. Levin and B. Nadler, “Natural image denoising: Optimality and inherent bounds,” in Conference on Computer Vision and Pattern Recognition (CVPR) , 2011, pp. 2833–2840
2011
Earlier work this paper cites.
P. Chatterjee and P. Milanfar, “Practical bounds on image denoising: From estimation to information,” IEEE Transactions on Image Processing (TIP) , vol. 20, no. 5, pp. 1221–1233, 2011
2011
Earlier work this paper cites.
T. M. Cover and J. A. Thomas, Elements of information theory . John Wiley & Sons, 2012
2012
Earlier work this paper cites.
A. Mittal, A. K. Moorthy, and A. C. Bovik, “No-reference image quality assessment in the spatial domain,” IEEE Transactions on Image Processing , vol. 21, no. 12, pp. 4695–4708, 2012
2012
Earlier work this paper cites.
M. A. Saad, A. C. Bovik, and C. Charrier, “Blind image quality assessment: A natural scene statistics approach in the DCT domain,” IEEE transactions on Image Processing , vol. 21, no. 8, pp. 3339–3352, 2012
2012
Earlier work this paper cites.
A. Levin, B. Nadler, F. Durand, and W. T. Freeman, “Patch complexity, finite pixel correlations and optimal denoising,” in European Conference on Computer Vision (ECCV) , 2012, pp. 73–86
2012
Earlier work this paper cites.
P. Ye, J. Kumar, L. Kang, and D. Doermann, “Unsupervised feature learning framework for no-reference image quality assessment,” in Conference on Computer Vision and Pattern Recognition (CVPR) , 2012, pp. 1098–1105
2012
Earlier work this paper cites.
F. Nielsen, “Hypothesis testing, information divergence and computational geometry,” in Geometric Science of Information , 2013, pp. 241–248
2013
Earlier work this paper cites.

Similar

A. Mittal, R. Soundararajan, and A. C. Bovik, “Making a “completely blind” image quality analyzer,” IEEE Signal Processing Letters , vol. 20, no. 3, pp. 209–212, 2013
2013
Cited alongside, same era.
Y. Liu, J. Wang, S. Cho, A. Finkelstein, and S. Rusinkiewicz, “A no-reference metric for evaluating the quality of motion deblurring.” ACM Transactions on Graphics (TOG) , vol. 32, no. 6, pp. 175–1, 2013
2013
Cited alongside, same era.
C.-Y. Yang, C. Ma, and M.-H. Yang, “Single-image super-resolution: A benchmark,” in European Conference on Computer Vision (ECCV) , 2014, pp. 372–386
2014
Cited alongside, same era.
I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in Advances in Neural Information Processing Systems (NIPS) , 2014, pp. 2672–2680
2014
Cited alongside, same era.
T. Van Erven and P. Harremos, “Rényi divergence and Kullback-Leibler divergence,” IEEE Transactions on Information Theory , vol. 60, no. 7, pp. 3797–3820, 2014
2014
Cited alongside, same era.
L. Kang, P. Ye, Y. Li, and D. Doermann, “Convolutional neural networks for no-reference image quality assessment,” in Conference on Computer Vision and Pattern Recognition (CVPR) , 2014, pp. 1733–1740
2014
Cited alongside, same era.
R. Timofte, V. De Smet, and L. Van Gool, “A+: Adjusted anchored neighborhood regression for fast super-resolution,” in Asian Conference on Computer Vision , 2014, pp. 111–126
2014
Cited alongside, same era.
C. Dong, C. C. Loy, K. He, and X. Tang, “Learning a deep convolutional network for image super-resolution,” in European Conference on Computer Vision (ECCV) , 2014, pp. 184–199
2014
Cited alongside, same era.
D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in International Conference on Learning Representations (ICLR) , 2014
2014
Cited alongside, same era.
E. L. Denton, S. Chintala, R. Fergus et al. , “Deep generative image models using a Laplacian pyramid of adversarial networks,” in Advances in Neural Information Processing Systems (NIPS) , 2015, pp. 1486–1494
2015
Cited alongside, same era.
J.-B. Huang, A. Singh, and N. Ahuja, “Single image super-resolution from transformed self-exemplars,” in Conference on Computer Vision and Pattern Recognition (CVPR) , 2015, pp. 5197–5206
2015
Cited alongside, same era.
J. Johnson, A. Alahi, and L. Fei-Fei, “Perceptual losses for real-time style transfer and super-resolution,” in European Conference on Computer Vision (ECCV) , 2016, pp. 694–711
2016
Cited alongside, same era.

Then

S. Guadarrama, R. Dahl, D. Bieber, M. Norouzi, J. Shlens, and K. Murphy, “PixColor: Pixel recursive colorization,” British Machine Vision Conference (BMVC) , 2017
2017
Closest in time.
O. Rippel and L. Bourdev, “Real-time adaptive image compression,” in International Conference on Machine Learning (ICML) , 2017, pp. 2922–2930
2017
Closest in time.
J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networks,” in International Conference on Computer Vision (ICCV) , 2017, pp. 2223–2232
2017
Closest in time.
M.-Y. Liu, T. Breuel, and J. Kautz, “Unsupervised image-to-image translation networks,” in Advances in neural information processing systems (NIPS) , 2017, pp. 700–708
2017
Closest in time.
M. Arjovsky, S. Chintala, and L. Bottou, “Wasserstein generative adversarial networks,” in International Conference on Machine Learning (ICML) , 2017, pp. 214–223
2017
Closest in time.
I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. C. Courville, “Improved training of wasserstein gans,” in Advances in Neural Information Processing Systems (NIPS) , 2017, pp. 5769–5779
2017
Closest in time.
C. Ma, C.-Y. Yang, X. Yang, and M.-H. Yang, “Learning a no-reference quality metric for single-image super-resolution,” Computer Vision and Image Understanding , vol. 158, pp. 1–16, 2017
2017
Closest in time.
W.-S. Lai, J.-B. Huang, N. Ahuja, and M.-H. Yang, “Deep Laplacian pyramid networks for fast and accurate super-resolution,” in Conference on Computer Vision and Pattern Recognition (CVPR) , 2017, pp. 624–632
2017
Closest in time.
W. Bae, J. Yoo, and J. Chul Ye, “Beyond deep residual learning for image restoration: Persistent homology-guided manifold simplification,” in Conference on Computer Vision and Pattern Recognition (CVPR) Workshops , 2017, pp. 145–153
2017
Closest in time.
B. Lim, S. Son, H. Kim, S. Nah, and K. M. Lee, “Enhanced deep residual networks for single image super-resolution,” in Conference on Computer Vision and Pattern Recognition (CVPR) Workshops , 2017
2017
Closest in time.
Y. Blau and T. Michaeli, “The perception-distortion tradeoff,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2018, pp. 6228–6237, doi: 10.1109/CVPR.2018.00652
2018
Closest in time.
R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” in Conference on Computer Vision and Pattern Recognition (CVPR) , 2018, pp. 586–595
2018
Closest in time.
X. Wang, K. Yu, S. Wu, J. Gu, Y. Liu, C. Dong, Y. Qiao, and C. Change Loy, “Esrgan: Enhanced super-resolution generative adversarial networks,” in Proceedings of the European Conference on Computer Vision (ECCV) workshops , 2018
2018
Closest in time.
J. Yu, Z. Lin, J. Yang, X. Shen, X. Lu, and T. S. Huang, “Generative image inpainting with contextual attention,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2018, pp. 5505–5514
2018
Closest in time.
E. Agustsson, M. Tschannen, F. Mentzer, R. Timofte, and L. Van Gool, “Generative adversarial networks for extreme learned image compression,” arXiv preprint arXiv:1804.02958 , 2018
Original
2018
Closest in time.
M. Tschannen, E. Agustsson, and M. Lucic, “Deep generative models for distribution-preserving lossy compression,” in Proc. Conference on Neural Information Processing Systems (NeurIPS) , 2018
2018
Closest in time.
O. Kupyn, V. Budzan, M. Mykhailych, D. Mishkin, and J. Matas, “Deblurgan: Blind motion deblurring using conditional adversarial networks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2018, pp. 8183–8192
2018
Closest in time.
S. Santurkar, D. Budden, and N. Shavit, “Generative compression,” in Proc. Picture Coding Symposium (PCS) , 2018
2018
Closest in time.
R. Matsumoto, “Introducing the perception-distortion tradeoff into the rate-distortion theory of general information sources,” IEICE Communications Express , vol. 7, no. 11, pp. 427–431, 2018
2018
Closest in time.
Y. Blau, R. Mechrez, R. Timofte, T. Michaeli, and L. Zelnik-Manor, “The 2018 pirm challenge on perceptual image super-resolution,” in Proceedings of the European Conference on Computer Vision (ECCV) , 2018
2018
Closest in time.
X. Deng, “Enhancing image quality via style transfer for single image super-resolution,” IEEE Signal Processing Letters , 2018
2018
Closest in time.
R. Mechrez, I. Talmi, F. Shama, and L. Zelnik-Manor, “Maintaining natural image statistics with the contextual loss,” in Asian Conference on Computer Vision . Springer, 2018, pp. 427–443
2018
Closest in time.
T. R. Shaham, T. Dekel, and T. Michaeli, “SinGAN: Learning a generative model from a single natural image,” in International Conference on Computer Vision (ICCV) , 2019
2019
Closest in time.
Y. Blau and T. Michaeli, “Rethinking lossy compression: The Rate-distortion-perception tradeoff,” in International Conference on Machine Learning (ICML) , 2019
2019
Closest in time.
R. Matsumoto, “Rate-distortion-perception tradeoff of variable-length source coding for general information sources,” IEICE Communications Express , vol. 8, no. 2, pp. 38–42, 2019
2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…