2016

Learning Representations for Automatic Colorization

Larsson, Gustav, Maire, Michael, Shakhnarovich, Gregory

Understand

We develop a fully automatic image colorization system.

Our approach leverages recent advances in deep networks, exploiting both low-level and semantic representations.
As many scene elements naturally appear according to multimodal color distributions, we train our model to predict per-pixel color histograms.
This intermediate output can be used to automatically generate a color image, or further manipulated prior to image formation.

Built on

Shi, J., Malik, J.: Normalized cuts and image segmentation. Pattern Analysis and Machine Intelligence, IEEE Transactions on 22(8) (2000)
2000
Earlier work this paper cites.
Welsh, T., Ashikhmin, M., Mueller, K.: Transferring color to greyscale images. ACM Transactions on Graphics (TOG) 21(3) (2002)
2002
Earlier work this paper cites.
Levin, A., Lischinski, D., Weiss, Y.: Colorization using optimization. ACM Transactions on Graphics (TOG) 23(3) (2004)
2004
Earlier work this paper cites.
Sỳkora, D., Buriánek, J., Žára, J.: Unsupervised colorization of black-and-white cartoons. In: International symposium on Non-photorealistic animation and rendering (2004)
2004
Earlier work this paper cites.
Huang, Y.C., Tung, Y.S., Chen, J.C., Wang, S.W., Wu, J.L.: An adaptive edge detection based colorization algorithm and its applications. In: ACM international conference on Multimedia (2005)
2005
Earlier work this paper cites.
Irony, R., Cohen-Or, D., Lischinski, D.: Colorization by example. In: Eurographics Symp. on Rendering (2005)
2005
Earlier work this paper cites.
Sapiro, G.: Inpainting the colors. In: ICIP (2005)
2005
Earlier work this paper cites.
Tai, Y.W., Jia, J., Tang, C.K.: Local color transfer via probabilistic segmentation by expectation-maximization. In: CVPR (2005)
2005
Earlier work this paper cites.
Qu, Y., Wong, T.T., Heng, P.A.: Manga colorization. ACM Transactions on Graphics (TOG) 25(3) (2006)
2006
Earlier work this paper cites.
Yatziv, L., Sapiro, G.: Fast image and video colorization using chrominance blending. Image Processing, IEEE Transactions on 15(5) (2006)
2006
Earlier work this paper cites.
Luan, Q., Wen, F., Cohen-Or, D., Liang, L., Xu, Y.Q., Shum, H.Y.: Natural image colorization. In: Eurographics conference on Rendering Techniques (2007)
2007
Earlier work this paper cites.
Charpiat, G., Hofmann, M., Schölkopf, B.: Automatic image colorization via multimodal predictions. In: ECCV (2008)
2008
Earlier work this paper cites.
Tola, E., Lepetit, V., Fua, P.: A fast local descriptor for dense matching. In: CVPR (2008)
2008
Earlier work this paper cites.
Morimoto, Y., Taguchi, Y., Naemura, T.: Automatic colorization of grayscale images using multiple images on the web. In: SIGGRAPH: Posters (2009)
2009
Earlier work this paper cites.
Charpiat, G., Bezrukov, I., Altun, Y., Hofmann, M., Schölkopf, B.: Machine learning methods for automatic image colorization. In: Computational Photography: Methods and Applications. CRC Press (2010)
2010
Earlier work this paper cites.
Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: AISTATS (2010)
2010
Earlier work this paper cites.

Similar

Xiao, J., Hays, J., Ehinger, K.A., Oliva, A., Torralba, A.: Sun database: Large-scale scene recognition from abbey to zoo. In: CVPR (2010)
2010
Cited alongside, same era.
Chia, A.Y.S., Zhuo, S., Gupta, R.K., Tai, Y.W., Cho, S.Y., Tan, P., Lin, S.: Semantic colorization with internet images. ACM Transactions on Graphics (TOG) 30(6) (2011)
2011
Cited alongside, same era.
Gupta, R.K., Chia, A.Y.S., Rajan, D., Ng, E.S., Zhiyong, H.: Image colorization using similar images. In: ACM international conference on Multimedia (2012)
2012
Cited alongside, same era.
Farabet, C., Couprie, C., Najman, L., LeCun, Y.: Learning hierarchical features for scene labeling. Pattern Analysis and Machine Intelligence, IEEE Transactions on 35(8) (2013)
2013
Cited alongside, same era.
Ganin, Y., Lempitsky, V.S.: N 4 {}^{\mbox{4}} -fields: Neural network nearest neighbor fields for image transforms. In: ACCV (2014)
2014
Cited alongside, same era.
Ion, A., Carreira, J., Sminchisescu, C.: Probabilistic joint image segmentation and labeling by figure-ground composition. International journal of computer vision 107(1), 40–57 (2014)
2014
Cited alongside, same era.
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., Darrell, T.: Caffe: Convolutional architecture for fast feature embedding. arXiv preprint arXiv:1408.5093 (2014)
Original
2014
Cited alongside, same era.
Maire, M., Yu, S.X., Perona, P.: Reconstructive sparse code transfer for contour detection and semantic labeling. In: ACCV (2014)
2014
Cited alongside, same era.
Patterson, G., Xu, C., Su, H., Hays, J.: The sun attribute database: Beyond categories for deeper scene understanding. International Journal of Computer Vision 108(1-2) (2014)
2014
Cited alongside, same era.
Tsaftaris, S.A., Casadio, F., Andral, J.L., Katsaggelos, A.K.: A novel visualization tool for art history and conservation: Automated colorization of black and white archival photographs of works of art. Studies in Conservation 59(3) (2014)
2014
Cited alongside, same era.
Bertasius, G., Shi, J., Torresani, L.: Deepedge: A multi-scale bifurcated deep network for top-down contour detection. In: CVPR (2015)
2015
Cited alongside, same era.
Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: Semantic image segmentation with deep convolutional nets and fully connected crfs. In: ICLR (2015)
2015
Cited alongside, same era.

Then

Hariharan, B., an R. Girshick, P.A., Malik, J.: Hypercolumns for object segmentation and fine-grained localization. CVPR (2015)
2015
Later among the works it cites.
Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: ICML (2015)
2015
Later among the works it cites.
Liu, W., Rabinovich, A., Berg, A.C.: Parsenet: Looking wider to see better. arXiv preprint arXiv:1506.04579 (2015)
Original
2015
Later among the works it cites.
Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: CVPR (2015)
2015
Later among the works it cites.
Mostajabi, M., Yadollahpour, P., Shakhnarovich, G.: Feedforward semantic segmentation with zoom-out features. In: CVPR (2015)
2015
Later among the works it cites.
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., Fei-Fei, L.: ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV) 115(3) (2015)
2015
Later among the works it cites.
Shen, W., Wang, X., Wang, Y., Bai, X., Zhang, Z.: Deepcontour: A deep convolutional feature learned by positive-sharing loss for contour detection. In: CVPR (2015)
2015
Later among the works it cites.
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: ICLR (2015)
2015
Later among the works it cites.
Xie, S., Tu, Z.: Holistically-nested edge detection. In: ICCV (2015)
2015
Later among the works it cites.
Dumoulin, V., Belghazi, I., Poole, B., Lamb, A., Arjovsky, M., Mastropietro, O., Courville, A.: Adversarially learned inference. arXiv preprint arXiv:1606.00704 (2016)
Original
2016
Closest in time.
Iizuka, S., Simo-Serra, E., Ishikawa, H.: Let there be Color!: Joint End-to-end Learning of Global and Local Image Priors for Automatic Image Colorization with Simultaneous Classification. ACM Transactions on Graphics (Proc. of SIGGRAPH 2016) 35(4) (2016)
2016
Closest in time.
Noroozi, M., Favaro, P.: Unsupervised learning of visual representations by solving jigsaw puzzles. In: ECCV (2016)
2016
Closest in time.
Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., Efros, A.A.: Context encoders: Feature learning by inpainting. In: CVPR (2016)
2016
Closest in time.
Zhang, R., Isola, P., Efros, A.A.: Colorful image colorization. In: ECCV (2016)
2016
Closest in time.
Donahue, J., Krähenbühl, P., Darrell, T.: Adversarial feature learning. In: ICLR (2017)
2017
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…