2017

SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability

Raghu, Maithra, Gilmer, Justin, Yosinski, Jason et al.

Understand

We propose a new technique, Singular Vector Canonical Correlation Analysis (SVCCA), a tool for quickly comparing two representations in a way that is both invariant to affine transform (allowing comparison between different layers and networks) and fast to compute (allowing more comparisons to be calculated than with previous methods).

We deploy this tool to measure the intrinsic dimensionality of layers, showing in some cases needless over-parameterization; to probe learning dynamics throughout training, finding that networks converge to final representations from the bottom up; to show where class-specific information in networks is formed; and to suggest new training regimes that simultaneously save computation and overfit less.
Code: https://github.com/google/svcca/

Built on

Matrix analysis
Roger A Horn and Charles R Johnson · 1985
Earlier work this paper cites.
Canonical correlation analysis: An overview with application to learning methods
D. R. Hardoon, S. Szedmak, and J. Shawe-Taylor · 2004
Earlier work this paper cites.
Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions
Nathan Halko, Martinsson Per-Gunnar, and Joel A. Tropp · 2011
Earlier work this paper cites.
Kernel analysis of deep networks
Grégoire Montavon, Mikio L Braun, and Klaus-Robert Müller · 2011
Earlier work this paper cites.
Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups
Geoffrey Hinton, Li Deng, Dong Yu, George E. Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N Sainath, et al · 2012
Earlier work this paper cites.
Imagenet classification with deep convolutional neural networks
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton · 2012
Earlier work this paper cites.
Understanding deep architectures using a recursive convolutional network
Original
David Eigen, Jason Rolfe, Rob Fergus, and Yann LeCun · 2013
Earlier work this paper cites.

Similar

Deep inside convolutional networks: Visualising image classification models and saliency maps
Original
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman · 2013
Cited alongside, same era.
Intriguing properties of neural networks
Original
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus · 2013
Cited alongside, same era.
Improving vector space word representations using multilingual correlation
Manaal Faruqui and Chris Dyer · 2014
Cited alongside, same era.
Visualizing and understanding convolutional networks
Matthew D Zeiler and Rob Fergus · 2014
Cited alongside, same era.
Object detectors emerge in deep scene cnns
Original
Bolei Zhou, Aditya Khosla, Àgata Lapedriza, Aude Oliva, and Antonio Torralba · 2014
Cited alongside, same era.
Deep residual learning for image recognition
Original
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2015
Cited alongside, same era.
Understanding image representations by measuring their equivariance and equivalence
Karel Lenc and Andrea Vedaldi · 2015
Cited alongside, same era.

Then

Convergent learning: Do different neural networks learn the same representations?
Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John Hopcroft · 2015
Later among the works it cites.
Understanding deep image representations by inverting them
Aravindh Mahendran and Andrea Vedaldi · 2015
Later among the works it cites.
A neural network that finds a naturalistic solution for the production of muscle activity
David Sussillo, Mark M Churchland, Matthew T Kaufman, and Krishna V Shenoy · 2015
Later among the works it cites.
Understanding neural networks through deep visualization
Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson · 2015
Later among the works it cites.
Understanding intermediate layers using linear classifier probes
Original
Guillaume Alain and Yoshua Bengio · 2016
Later among the works it cites.
Convergent Learning: Do different neural networks learn the same representations?
Y. Li, J. Yosinski, J. Clune, H. Lipson, and J. Hopcroft · 2016
Later among the works it cites.
Google’s neural machine translation system: Bridging the gap between human and machine translation
Original
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al · 2016
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…