2015

Super-Resolution with Deep Convolutional Sufficient Statistics

Bruna, Joan, Sprechmann, Pablo, LeCun, Yann

Understand

Inverse problems in image and audio, and super-resolution in particular, can be seen as high-dimensional structured prediction problems, where the goal is to characterize the conditional distribution of a high-resolution output given its low-resolution corrupted observation.

When the scaling ratio is small, point estimates achieve impressive performance, but soon they suffer from the regression-to-the-mean problem, result of their inability to capture the multi-modality of this conditional distribution.
Modeling high-dimensional image and audio distributions is a hard task, requiring both the ability to model complex geometrical structures and textured regions.
In this paper, we propose to use as conditional model a Gibbs distribution, where its sufficient statistics are given by deep convolutional neural networks.

Built on

Gradient-based learning applied to document recognition
LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P · 1998
Earlier work this paper cites.
Filters, random fields and maximum entropy (frame): Towards a unified theory for texture modeling
Zhu, S. C., Wu, Y, and Mumford, D · 1998
Earlier work this paper cites.
A parametric texture model based on joint statistics of complex wavelet coefficients
Portilla, Javier and Simoncelli, Eero P · 2000
Earlier work this paper cites.
Example-based super-resolution
Freeman, W T, Jones, T R, and Pasztor, E C · 2002
Earlier work this paper cites.
Image denoising via sparse and redundant representations over learned dictionaries
Elad, M. and Aharon, M · 2006
Earlier work this paper cites.
A tutorial on energy-based learning
LeCun, Yann, Chopra, Sumit, Hadsell, Raia, Ranzato, M, and Huang, F · 2006
Earlier work this paper cites.
Image super-resolution as sparse representation of raw image patches
Yang, J., Wright, J., Huang, T., and Ma, Y · 2008
Earlier work this paper cites.
Imagenet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L · 2009
Earlier work this paper cites.
Measuring invariances in deep networks
Goodfellow, I., Le, Q., Saxe, A., Lee, H., and Ng, A. Y · 2009
Earlier work this paper cites.
Online dictionary learning for sparse coding
Mairal, J., Bach, F., Ponce, J., and Sapiro, G · 2009
Earlier work this paper cites.
Learning fast approximations of sparse coding
Gregor, K. and LeCun, Y · 2010
Earlier work this paper cites.
Learning deep energy models
Ngiam, Jiquan, Chen, Zhenghao, Koh, Pang W, and Ng, Andrew Y · 2011
Earlier work this paper cites.

Similar

Task-driven dictionary learning
Mairal, J., Bach, F., and Ponce, J · 2012
Cited alongside, same era.
Group invariant scattering
Mallat, S · 2012
Cited alongside, same era.
Coupled dictionary training for image super-resolution
Yang, J., Wang, Z., Lin, Z., Cohen, S., and Huang, T. S · 2012
Cited alongside, same era.
Deep generative stochastic networks trainable by backprop
Bengio, Y., Thibodeau-Laufer, E., Alain, G., and Yosinski, J · 2013
Cited alongside, same era.
Invariant scattering convolution networks
Bruna, J. and Mallat, S · 2013
Cited alongside, same era.
Auto-encoding variational bayes
Original
Kingma, D. P and Welling, M · 2013
Cited alongside, same era.
Signal recovery from pooling representations
Bruna, J., Szlam, A., and LeCun, Y · 2014
Cited alongside, same era.
Deep network cascade for image super-resolution
Cui, Z., Chang, H., Shan, S., Zhong, B., and Chen, X · 2014
Cited alongside, same era.
Learning a deep convolutional network for image super-resolution
Dong, C., Loy, Chen C., He, K., and Tang, X · 2014
Cited alongside, same era.
Generative adversarial nets
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y · 2014
Cited alongside, same era.
The local low-dimensionality of natural images
Original
Hénaff, O. J., Ballé, J., Rabinowitz, N. C., and Simoncelli, E. P · 2014
Cited alongside, same era.
Adam: A method for stochastic optimization
Original
Kingma, Diederik and Ba, Jimmy · 2014
Cited alongside, same era.

Then

Learning and transferring mid-level image representations using convolutional neural networks
Oquab, M., Bottou, L., Laptev, I., and Sivic, J · 2014
Later among the works it cites.
Very deep convolutional networks for large-scale image recognition
Original
Simonyan, K. and Zisserman, A · 2014
Later among the works it cites.
Generative modeling of convolutional neural networks
Dai, Jifeng, Lu, Yang, and Wu, Ying Nian · 2015
Closest in time.
Deep generative image models using a laplacian pyramid of adversarial networks
Original
Denton, E., Chintala, S., Szlam, A., and Fergus, R · 2015
Closest in time.
Texture synthesis and the controlled generation of natural stimuli using convolutional neural networks
Original
Gatys, Leon A, Ecker, Alexander S, and Bethge, Matthias · 2015
Closest in time.
DRAW: A recurrent neural network for image generation
Original
Gregor, K., Danihelka, I., Graves, A., and Wierstra, D · 2015
Closest in time.
Variational inference with normalizing flows
Original
Jimenez-Rezende, D and Mohamed, S · 2015
Closest in time.
Deep unsupervised learning using nonequilibrium thermodynamics
Original
Sohl-Dickstein, J, Weiss, E A, Maheswaranathan, N, and Ganguli, S · 2015
Closest in time.
Learning efficient sparse and low rank models
Sprechmann, P., Bronstein, A.M., and Sapiro, G · 2015
Closest in time.
Discriminative regularization for generative models
Original
Lamb, Alex, Dumoulin, Vincent, and Courville, Aaron · 2016
Closest in time.
A theory of generative convnet
Xie, J., Lu, Y., Zhu, S., and Wu, Y · 2016
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…