2018

Exploiting the Potential of Standard Convolutional Autoencoders for Image Restoration by Evolutionary Search

Suganuma, Masanori, Ozay, Mete, Okatani, Takayuki

Understand

Researchers have applied deep neural networks to image restoration tasks, in which they proposed various network architectures, loss functions, and training methods.

In particular, adversarial training, which is employed in recent studies, seems to be a key ingredient to success.
In this paper, we show that simple convolutional autoencoders (CAEs) built upon only standard network components, i.e., convolutional layers and skip connections, can outperform the state-of-the-art methods which employ adversarial training and sophisticated loss functions.
The secret is to employ an evolutionary algorithm to automatically search for good architectures.

Built on

Combinations of genetic algorithms and neural networks: A survey of the state of the art
Schaffer, J. D., Whitley, D., and Eshelman, L. J · 1992
Earlier work this paper cites.
Gradient-based learning applied to document recognition
LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P · 1998
Earlier work this paper cites.
Cartesian genetic programming
Miller, J. F. and Thomson, P · 2000
Earlier work this paper cites.
A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics
Martin, D., Fowlkes, C., Tal, D., and Malik, J · 2001
Earlier work this paper cites.
Evolving neural networks through augmenting topologies
Stanley, K. O. and Miikkulainen, R · 2002
Earlier work this paper cites.
Introduction to Evolutionary Computing , volume 53
Eiben, A. E. and Smith, J. E · 2003
Earlier work this paper cites.
k-SVD: An algorithm for designing overcomplete dictionaries for sparse representation
Aharon, M., Elad, M., and Bruckstein, A · 2006
Earlier work this paper cites.
Redundancy and computational efficiency in cartesian genetic programming
Miller, J. F. and Smith, S. L · 2006
Earlier work this paper cites.
Image upsampling via imposed edge statistics
Fattal, R · 2007
Earlier work this paper cites.
BM3d image denoising with shape-adaptive principal component analysis
Dabov, K., Foi, A., Katkovnik, V., and Egiazarian, K · 2009
Earlier work this paper cites.
Rectified linear units improve restricted boltzmann machines
Nair, V. and Hinton, G. E · 2010
Earlier work this paper cites.
Image super-resolution via sparse representation
Yang, J., Wright, J., Huang, T. S., and Ma, Y · 2010
Earlier work this paper cites.
Reading digits in natural images with unsupervised feature learning
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., and Ng, A. Y · 2011
Earlier work this paper cites.
Imagenet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., and Hinton, G. E · 2012
Earlier work this paper cites.
Image denoising and inpainting with deep neural networks
Xie, J., Xu, L., and Chen, E · 2012
Earlier work this paper cites.
3d object representations for fine-grained categorization
Krause, J., Stark, M., Deng, J., and Fei-Fei, L · 2013
Earlier work this paper cites.
Learning a deep convolutional network for image super-resolution
Dong, C., Loy, C. C., He, K., and Tang, X · 2014
Earlier work this paper cites.
Depth map prediction from a single image using a multi-scale deep network
Eigen, D., Puhrsch, C., and Fergus, R · 2014
Earlier work this paper cites.

Similar

Generative adversarial nets
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y · 2014
Cited alongside, same era.
Total variation blind deconvolution: The devil is in the details
Perrone, D. and Favaro, P · 2014
Cited alongside, same era.
Deep convolutional neural network for image deconvolution
Xu, L., Ren, J. SJ., Liu, C., and Jia, J · 2014
Cited alongside, same era.
Adam: A method for stochastic optimization
Kingma, D. P. and Ba, J. L · 2015
Cited alongside, same era.
Deep learning face attributes in the wild
Liu, Z., Luo, P., Wang, X., and Tang, X · 2015
Cited alongside, same era.
Training very deep networks
Srivastava, R. K., Greff, K., and Schmidhuber, J · 2015
Cited alongside, same era.
Deep residual learning for image recognition
He, K., Zhang, X., Ren, S., and Sun, J · 2016
Cited alongside, same era.
Perceptual losses for real-time style transfer and super-resolution
Johnson, J., Alahi, A., and Fei-Fei, L · 2016
Cited alongside, same era.
Reconnet: Non-iterative reconstruction of images from compressively sensed measurements
Kulkarni, K., Lohit, S., Turaga, P., Kerviche, R., and Ashok, A · 2016
Cited alongside, same era.
Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections
Mao, X., Shen, C., and Yang, Y · 2016
Cited alongside, same era.
Context encoders: Feature learning by inpainting
Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., and Efros, A. A · 2016
Cited alongside, same era.
Colorful image colorization
Zhang, R., Isola, P., and Efros, A. A · 2016
Cited alongside, same era.

Then

Progressive neural architecture search
Original
Liu, C., Zoph, B., Shlens, J., Hua, W., Li, L., Fei-Fei, L., Yuille, A., Huang, J., and Murphy, K · 2017
Later among the works it cites.
Are gans created equal? a large-scale study
Original
Lucic, M., Kurach, K., Michalski, M., Gelly, S., and Bousquet, O · 2017
Later among the works it cites.
Evolving deep neural networks
Miikkulainen, R., Liang, J. Z., Meyerson, E., Rawal, A., Fink, D., Francon, O., Raju, B., Shahrzad, H., Navruzyan, A., Duffy, N., and Hodjat, B · 2017
Later among the works it cites.
Learning to invert: Signal recovery via deep convolutional networks
Mousavi, A. and Baraniuk, R. G · 2017
Later among the works it cites.
Automatic differentiation in pytorch
Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., and Lerer, A · 2017
Later among the works it cites.
Large-scale evolution of image classifiers
Real, E., Moore, S., Selle, A., Saxena, S., Suematsu, Y. L., Le, Q. V., and Kurakin, A · 2017
Later among the works it cites.
A genetic programming approach to designing convolutional neural network architectures
Suganuma, M., Shirakawa, S., and Nagao, T · 2017
Later among the works it cites.
Memnet: A persistent memory network for image restoration
Tai, Y., Yang, J., Liu, X., and Xu, C · 2017
Later among the works it cites.
Genetic CNN
Xie, L. and Yuille, A. L · 2017
Later among the works it cites.
Multi-scale continuous crfs as sequential deep networks for monocular depth estimation
Xu, D., Ricci, E., Ouyang, W., Wang, X., and Sebe, N · 2017
Later among the works it cites.
High-resolution image inpainting using multi-scale neural patch synthesis
Yang, C., Lu, X., Lin, Z., Shechtman, E., Wang, O., and Li, H · 2017
Later among the works it cites.
Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising
Zhang., K., Zuo, W., Chen, Y., Meng, D., and Zhang, L · 2017
Later among the works it cites.
Practical network blocks design with Q-Learning
Zhong, Z., Yan, J., and Liu, C · 2017
Later among the works it cites.
Neural architecture search with reinforcement learning
Zoph, B. and Le, Q. V · 2017
Later among the works it cites.
Learning transferable architectures for scalable image recognition
Original
Zoph, B., Vasudevan, V., Shlens, J., and Le, Q. V · 2017
Later among the works it cites.
Smash: one-shot model architecture search through hypernetworks
Brock, A., Lim, T., Ritchie, J. M., and Weston, N · 2018
Closest in time.
Hierarchical representations for efficient architecture search
Liu, H., Simonyan, K., Vinyals, O., Fernando, C., and Kavukcuoglu, K · 2018
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…