2020

Rethinking Image Inpainting via a Mutual Encoder-Decoder with Feature Equalizations

Liu, Hongyu, Jiang, Bin, Song, Yibing et al.

Understand

Deep encoder-decoder based CNNs have advanced image inpainting methods for hole filling.

While existing methods recover structures and textures step-by-step in the hole regions, they typically use two encoder-decoders for separate recovery.
The CNN features of each encoder are learned to capture either missing structures or textures without considering them as a whole.
The insufficient utilization of these encoder features limit the performance of recovering both structures and textures.

Built on

Tomasi, C., Manduchi, R.: Bilateral filtering for gray and color images. In: CVPR (1998)
1998
Earlier work this paper cites.
Bertalmio, M., Sapiro, G., Caselles, V., Ballester, C.: Image inpainting. In: SIGGRAPH (2000)
2000
Earlier work this paper cites.
Ballester, C., Bertalmio, M., Caselles, V., Sapiro, G., Verdera, J.: Filling-in by joint interpolation of vector fields and gray levels. TIP (2001)
2001
Earlier work this paper cites.
Efros, A., Freeman, W.: Image quilting for texture synthesis and transfer. In: SIGGRAPH (2001)
2001
Earlier work this paper cites.
Efros, A., Freeman, W.: Texture synthesis by nonparametric sampling. In: ICCV (2001)
2001
Earlier work this paper cites.
Levin, A., Zomet, A., Weiss, Y.: Learning how to inpaint from global image statistics. In: ICCV (2003)
2003
Earlier work this paper cites.
Criminisi, A., Pérez, P., Toyama, K.: Region filling and object removal by exemplar-based image inpainting. TIP (2004)
2004
Earlier work this paper cites.
Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. TIP (2004)
2004
Earlier work this paper cites.
Barnes, C., Shechtman, E., Finkelstein, A., Goldman, D.: Patchmatch: A randomized correspondence algorithm forstructural image editing. In: SIGGRAPH (2009)
2009
Earlier work this paper cites.
Xu, Z., Sun, J.: Image inpainting by patch propagation using patch sparsity. TIP (2010)
2010
Earlier work this paper cites.
Darabi, S., Shechtman, E., Barnes, C., Goldman, D.B., Sen, P.: Image melding: combining inconsistent images using patch-based synthesis. ACM Transactions on Graphics (2012)
2012
Earlier work this paper cites.
Xu, L., Yan, Q., Xia, Y., Jia, J.: Structure extraction from texture via relative total variation. SIGGRAPH (2012)
2012
Earlier work this paper cites.
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: NIPS (2014)
2014
Earlier work this paper cites.
Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: arXiv preprint arXiv:1412.6980 (2014)
Original
2014
Earlier work this paper cites.
Doersch, C., Singh, S., Gupta, A., Sivic, J., Efros, A.A.: What makes paris look like paris? Communications of the ACM (2015)
2015
Earlier work this paper cites.

Similar

Liu, Z., LuoPi, n., Wang, X., Tang, X.: Deep learning face attributes in the wild. In: ICCV (2015)
2015
Cited alongside, same era.
Yu, F., Koltun, V.: Multi-scale context aggregation by dilated convolutions. In: arXiv preprint arXiv:1511.07122 (2015)
Original
2015
Cited alongside, same era.
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR (2016)
2016
Cited alongside, same era.
Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: ECCV (2016)
2016
Cited alongside, same era.
Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., Efros, A.: Context encoders: Feature learning by inpainting. In: CVPR (2016)
2016
Cited alongside, same era.
Yeh, R., Chen, C., Lim, T., Johnson, M.H., N, D.: Semantic image inpainting with perceptual and contextual losses. In: arXiv preprint arXiv:1607.07539 (2016)
Original
2016
Cited alongside, same era.
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: Gans trained by a two time-scale update rule converge to a local nash equilibrium. In: NIPS (2017)
2017
Cited alongside, same era.
Iizuka, S., Simo-Serra, E., Ishikawa, H.: Globally and locally consistent image completion. In: SIGGRAPH (2017)
2017
Cited alongside, same era.
Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: CVPR (2017)
2017
Cited alongside, same era.
Li, Y., Liu, S., Yang, J., Yang, M.H.: Generative face completion. In: CVPR (2017)
2017
Cited alongside, same era.
Song, Y., Bao, L., He, S., Yang, Q., Yang, M.H.: Stylizing face images via multiple exemplars. CVIU (2017)
2017
Cited alongside, same era.
Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., Torralba, A.: Places: A 10 million image database for scene recognition. PAMI (2017)
2017
Cited alongside, same era.

Then

Liu, G., Reda, F.A., Shih, K.J., Wang, T.C., Tao, A., Catanzaro, B.: Image inpainting for irregular holes using partial convolutions. In: ECCV (2018)
2018
Later among the works it cites.
Miyato, T., Kataoka, T., Koyama, M., Yoshida, Y.: Spectral normalization for generative adversarial networks. In: arXiv preprint arXiv:1802.05957 (2018)
Original
2018
Later among the works it cites.
Song, Yuhangand Yang, C., Lin, Z., Liu, X., Huang, Q., Li, H., Jay, C.: Contextual-based image inpainting: Infer, match, and translate. In: ECCV (2018)
2018
Later among the works it cites.
Song, Y., Yang, C., Shen, Y., Wang, P., Huang, Q., Kuo, J.: Spg-net: Segmentation prediction and guidance network for image inpainting. In: arXiv preprint arXiv:1805.03356 (2018)
Original
2018
Later among the works it cites.
Wang, X., Girshick, R., Gupta, A., He, K.: Non-local neural networks. In: CVPR (2018)
2018
Later among the works it cites.
Wang, Y., Tao, X., Qi, X., Shen, X., Jia, J.: Image inpainting via generative multi-column convolutional neural networks. In: NIPS (2018)
2018
Later among the works it cites.
Yan, Z., Li, X., Li, M., Zuo, W., Shan, S.: Shift-net: Image inpainting via deep feature rearrangement. In: ECCV (2018)
2018
Later among the works it cites.
Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., Huang, T.S.: Generative image inpainting with contextual attention. In: CVPR (2018)
2018
Later among the works it cites.
Liu, H., Jiang, B., Xiao, Y., Yang, C.: Coherent semantic attention for image inpainting. In: ICCV (2019)
2019
Later among the works it cites.
Nazeri, K., Ng, E., Joseph, T., Qureshi, F.Z., Ebrahimi, M.: Edgeconnect: Generative image inpainting with adversarial edge learning. In: ICCV Workshops (2019)
2019
Later among the works it cites.
Ren, Y., Yu, X., Zhang, R., Li, T.H., Liu, S., Li, G.: Structureflow: Image inpainting via structure-aware appearance flow. In: ICCV (2019)
2019
Later among the works it cites.
Xiong, W., Yu, J., Lin, Z., Yang, J., Lu, X., Barnes, C., Luo, J.: Foreground-aware image inpainting. In: CVPR (2019)
2019
Later among the works it cites.
Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., Huang, T.S.: Free-form image inpainting with gated convolution. In: ICCV (2019)
2019
Later among the works it cites.
Zeng, Y., Fu, J., Chao, H., Guo, B.: Learning pyramid-context encoder network for high-quality image inpainting. In: CVPR (2019)
2019
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…