2015

Visualizing Deep Convolutional Neural Networks Using Natural Pre-Images

Mahendran, Aravindh, Vedaldi, Andrea

Understand

Image representations, from SIFT and bag of visual words to Convolutional Neural Networks (CNNs) are a crucial component of almost all computer vision systems.

However, our understanding of them remains limited.
In this paper we study several landmark representations, both shallow and deep, by a number of complementary visualization techniques.
These visualizations are based on the concept of "natural pre-image", namely a natural-looking image whose representation has some notable property.

Built on

Nature 290
Julesz, B.: Textons, the elements of texture perception, and their interactions · 1981
Earlier work this paper cites.
In: Proc. CogSci (1986)
Williams, R.J.: Inverting a connectionist network mapping by back-propagation of error · 1986
Earlier work this paper cites.
In: Proc. Int. Conf. on Neural Networks (1989)
Linden, A., Kindermann, J.: Inversion of multilayer nets · 1989
Earlier work this paper cites.
IEEE Trans. on Neural Networks 5
Lee, S., Kil, R.M.: Inverse mapping of continuous functions using local and global information · 1994
Earlier work this paper cites.
Clarendon Press, Oxford (1995)
Bishop, C.M.: Neural Networks for Pattern Recognition · 1995
Earlier work this paper cites.
ACM Transactions on Mathematical Software (TOMS) 23
Zhu, C., Byrd, R.H., Lu, P., Nocedal, J.: Algorithm 778: L-bfgs-b: Fortran subroutines for large-scale bound-constrained optimization · 1997
Earlier work this paper cites.
In: Proc. CVPR (1999)
Huang, W.J., Mumford, D.: Statistics of natural images and models · 1999
Earlier work this paper cites.
Proc. of the IEEE 87
Jensen, C.A., Reed, R.D., Marks, R.J., El-Sharkawi, M., Jung, J.B., Miyamoto, R., Anderson, G., Eggen, C.: Inversion of feedforward neural networks: algorithms and applications · 1999
Earlier work this paper cites.
In: ICCV (1999)
Lowe, D.G.: Object recognition from local scale-invariant features · 1999
Earlier work this paper cites.
IEEE Trans. on Neural Networks 10
Lu, B.L., Kita, H., Nishikawa, Y.: Inverting feedforward neural networks using linear and nonlinear programming · 1999
Earlier work this paper cites.
IJCV (2000)
Portilla, J., Simoncelli, E.P.: A parametric texture model based on joint statistics of complex wavelet coefficients · 2000
Earlier work this paper cites.
In: ICCV (2003)
Sivic, J., Zisserman, A.: Video Google: A text retrieval approach to object matching in videos · 2003
Earlier work this paper cites.
In: Proc. ECCV Workshop on Stat. Learn. in Comp. Vision (2004)
Csurka, G., Dance, C.R., Dan, L., Willamowski, J., Bray, C.: Visual categorization with bags of keypoints · 2004
Earlier work this paper cites.
In: CVPR (2005)
Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection · 2005
Earlier work this paper cites.
In: IEEE Int. Conf. on Fuzzy Systems (2005)
Várkonyi-Kóczy, A.R., Rövid, A.: Observer based iterative neural network model inversion · 2005
Earlier work this paper cites.
Science 313
Hinton, G.E., Salakhutdinov, R.R.: Reducing the dimensionality of data with neural networks · 2006
Earlier work this paper cites.
In: ECCV (2006)
Nowak, E., Jurie, F., Triggs, B.: Sampling strategies for bag-of-features image classification · 2006
Earlier work this paper cites.

Similar

In: CVPR (2006)
Perronnin, F., Dance, C.: Fisher kernels on visual vocabularies for image categorizaton · 2006
Cited alongside, same era.
Tech. Rep. 070012, UCLA CSD (2007)
Vedaldi, A.: An open implementation of the SIFT detector and descriptor · 2007
Cited alongside, same era.
Tech. Rep. 1341, University of Montreal (2009)
Erhan, D., Bengio, Y., Courville, A., Vincent, P.: Visualizing higher-layer features of a deep network · 2009
Cited alongside, same era.
In: NIPS (2009)
Krishnan, D., Fergus, R.: Fast image deconvolution using hyper-laplacian priors · 2009
Cited alongside, same era.
http://www.pascal-network.org/challenges/VOC/voc2010/workshop/index.html
Everingham, M., Van Gool, L., Williams, C.K.I., Winn, J., Zisserman, A.: The PASCAL Visual Object Classes Challenge 2010 (VOC2010) Results · 2010
Cited alongside, same era.
IEEE Transactions on Pattern Analysis and Machine Intelligence 32
Felzenszwalb, P.F., Girshick, R.B., McAllester, D., Ramanan, D.: Object detection with discriminatively trained part based models · 2010
Cited alongside, same era.
In: CVPR (2010)
Jégou, H., Douze, M., Schmid, C., Pérez, P.: Aggregating local descriptors into a compact image representation · 2010
Cited alongside, same era.
In: CVPR (2010)
Wang, J., Yang, J., Yu, K., Lv, F., Huang, T., Gong, Y.: Locality-constrained linear coding for image classification · 2010
Cited alongside, same era.
In: CVPR (2010)
Yang, J., Yu, K., Huang, T.: Supervised translation-invariant sparse coding · 2010
Cited alongside, same era.
In: ECCV (2010)
Zhou, X., Yu, K., Zhang, T., Huang, T.S.: Image classification using super-vector coding of local image descriptors · 2010
Cited alongside, same era.
Journal of Machine Learninig Research 12
Duchi, J., Hazan, E., Singer, Y.: Adaptive subgradient methods for online learning and stochastic optimization · 2011
Cited alongside, same era.
In: ICCV Workshop (2011)
Tatu, A., Lauze, F., Nielsen, M., Kimia, B.: Exploring the representation capabilities of the HOG descriptor · 2011
Cited alongside, same era.

Then

In: ICCV (2013)
Vondrick, C., Khosla, A., Malisiewicz, T., Torralba, A.: HOGgles: Visualizing object detection features · 2013
Later among the works it cites.
In: Proc. BMVC (2014)
Chatfield, K., Simonyan, K., Vedaldi, A., Zisserman, A.: Return of the devil in the details: Delving deep into convolutional nets · 2014
Later among the works it cites.
In: Proc. of Computer Vision Winter Workshop (2014)
Chen, Y., Ranftl, R., Pock, T.: A bi-level view of inpainting-based image compression · 2014
Later among the works it cites.
In: CVPR (2014)
Kato, H., Harada, T.: Image reconstruction from bag-of-visual-words · 2014
Later among the works it cites.
In: CoRR, vol. abs/1312.6229 (2014)
Original
Sermanet, P., Eigen, D., Zhang, X., Mathieu, M., Fergus, R., LeCun, Y.: Overfeat: Integrated recognition, localization and detection using convolutional networks · 2014
Later among the works it cites.
In: Proc. ICLR (2014)
Simonyan, K., Vedaldi, A., Zisserman, A.: Deep inside convolutional networks: Visualising image classification models and saliency maps · 2014
Later among the works it cites.
In: Proc. ICLR (2014)
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I.J., Fergus, R.: Intriguing properties of neural networks · 2014
Later among the works it cites.
http://www.vlfeat.org/matconvnet/ (2014)
Vedaldi, A., Lenc, K.: MatConvNet: CNNs for MATLAB · 2014
Later among the works it cites.
In: ECCV (2014)
Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional networks · 2014
Later among the works it cites.
CoRR (abs/1506.02753) (2015)
Original
Dosovitskiy, A., Brox, T.: Inverting convolutional networks with convolutional networks · 2015
Closest in time.
CoRR (2015)
Gatys, L.A., Ecker, A.S., Bethge, M.: A neural algorithm of artistic style · 2015
Closest in time.
In: Proc. NIPS (2015)
Gatys, L.A., Ecker, A.S., Bethge, M.: Texture synthesis and the controlled generation of natural stimuli using convolutional neural networks · 2015
Closest in time.
In: Proc. CVPR (2015)
Mahendran, A., Vedaldi, A.: Understanding deep image representations by inverting them · 2015
Closest in time.
URL http://googleresearch.blogspot.co.uk/2015/06/inceptionism-going-deeper-into-neural.html
Mordvintsev, A., Olah, C., Tyka, M.: Inceptionism: Going deeper into neural networks (2015) · 2015
Closest in time.
In: Proc. CVPR (2015)
Nguyen, A., Yosinski, J., Clune, J.: Deep neural networks are easily fooled: High confidence predictions for unrecognizable images · 2015
Closest in time.
IJCV 115
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., Fei-Fei, L.: ImageNet Large Scale Visual Recognition Challenge · 2015
Closest in time.
In: Proc. ICML Workshop (2015)
Yosinksi, J., Clune, J., Nguyen, A., Fuchs, T., Lipson, H.: Understanding neural networks through deep visualization · 2015
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…