Fetching the paper…

Reading the bibliography…

2019

Intrinsic dimension of data representations in deep neural networks

Ansuini, Alessio, Laio, Alessandro, Macke, Jakob H. et al.

Original
PDF
Post on X

Understand

Deep neural networks progressively transform their inputs across multiple processing layers.

What are the geometrical properties of the representations learned by these networks? Here we study the intrinsic dimensionality (ID) of data-representations, i.e.
the minimal number of parameters needed to describe a representation.
We find that, in a trained network, the ID is orders of magnitude smaller than the number of units in each layer.

Intrinsic dimension of data representations in deep neural networks · Around

Built on

Y. LeCun, J. S. Denker, and S. A. Solla, “Optimal brain damage,” in Advances in neural information processing systems
1990
Earlier work this paper cites.
E. P. Simoncelli and B. A. Olshausen, “Natural image statistics and neural representation,” Annual review of neuroscience
2001
Earlier work this paper cites.
B. A. Olshausen and D. J. Field, “Sparse coding of sensory inputs,” Current opinion in neurobiology
2004
Earlier work this paper cites.
E. Levina and P. J. Bickel, “Maximum likelihood estimation of intrinsic dimension,” in Advances in neural information processing systems
2005
Earlier work this paper cites.
J. J. DiCarlo and D. D. Cox, “Untangling invariant object recognition,” Trends in cognitive sciences
2007
Earlier work this paper cites.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “ImageNet: A Large-Scale Hierarchical Image Database,” in CVPR09
2009
Earlier work this paper cites.
Y. LeCun and C. Cortes, “MNIST handwritten digit database,” 2010
2010
Earlier work this paper cites.
A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Advances in neural information processing systems
2012
Earlier work this paper cites.
J. J. DiCarlo, D. Zoccolan, and N. C. Rust, “How does the brain solve visual object recognition?,” Neuron
2012
Earlier work this paper cites.
M. Denil, B. Shakibi, L. Dinh, M. Ranzato, and N. De Freitas, “Predicting parameters in deep learning,” in Advances in Neural Information Processing Systems
2013
Earlier work this paper cites.
K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556
Original
2014
Earlier work this paper cites.
L. Amsaleg, O. Chelly, T. Furon, S. Girard, M. E. Houle, K.-i. Kawarabayashi, and M. Nett, “Estimating local intrinsic dimensionality,” in Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining
2015
Earlier work this paper cites.

Similar

K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” CoRR
Original
2015
Cited alongside, same era.
C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals, “Understanding deep learning requires rethinking generalization,” CoRR
Original
2016
Cited alongside, same era.
R. Basri and D. Jacobs, “Efficient representation of low-dimensional manifolds using deep networks,” arXiv preprint arXiv:1602.04723
Original
2016
Cited alongside, same era.
P. P. Brahma, D. Wu, and Y. She, “Why deep learning works: A manifold disentanglement perspective.,” IEEE Trans. Neural Netw. Learning Syst
2016
Cited alongside, same era.
S. Chung, D. D. Lee, and H. Sompolinsky, “Linear readout of object manifolds,” Physical Review E
2016

Then

A. K. Lampinen and S. Ganguli, “An analytic theory of generalization dynamics and transfer learning in deep linear networks,” arXiv preprint arXiv:1809.10374
Original
2018
Later among the works it cites.
H. Huang, “Mechanisms of dimensionality reduction and decorrelation in deep neural networks,” Physical Review E
2018
Later among the works it cites.
X. Ma, B. Li, Y. Wang, S. M. Erfani, S. N. R. Wijewickrema, M. E. Houle, G. Schoenebeck, D. Song, and J. Bailey, “Characterizing adversarial subspaces using local intrinsic dimensionality,” CoRR
Original
2018
Later among the works it cites.
T. Yu, H. Long, and J. E. Hopcroft, “Curvature-based comparison of two neural networks,” arXiv preprint arXiv:1801.06801
Original
2018
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Cited alongside, same era.

L. Amsaleg, J. Bailey, D. Barbe, S. Erfani, M. E. Houle, V. Nguyen, and M. Radovanović, “The vulnerability of learning to adversarial perturbation increases with intrinsic dimensionality,” in Information Forensics and Security (WIFS), 2017 IEEE Workshop on

2017

Cited alongside, same era.

M. Raghu, J. Gilmer, J. Yosinski, and J. Sohl-Dickstein, “Svcca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability,” in Advances in Neural Information Processing Systems

2017

Cited alongside, same era.

E. Facco, M. d’Errico, A. Rodriguez, and A. Laio, “Estimating the intrinsic dimension of datasets by a minimal neighborhood information,” Scientific reports

2017

Cited alongside, same era.

S. Tafazoli, H. Safaai, G. De Franceschi, F. B. Rosselli, W. Vanzella, M. Riggi, F. Buffolo, S. Panzeri, and D. Zoccolan, “Emergence of transformation-tolerant representations of visual objects in rat lateral extrastriate cortex,” Elife

2017

Cited alongside, same era.

R. Shwartz-Ziv and N. Tishby, “Opening the black box of deep neural networks via information,” arXiv preprint arXiv:1703.00810

2017

Cited alongside, same era.

A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer, “Automatic differentiation in pytorch,” in NIPS-W

2017

Cited alongside, same era.

B. Neyshabur, Z. Li, S. Bhojanapalli, Y. LeCun, and N. Srebro, “Towards understanding the role of over-parametrization in generalization of neural networks,” arXiv preprint arXiv:1805.12076

2018

Cited alongside, same era.

A. Morcos, M. Raghu, and S. Bengio, “Insights on representational similarity in neural networks with canonical correlation,” in Advances in Neural Information Processing Systems

2018

Later among the works it cites.

X. Ma, Y. Wang, M. E. Houle, S. Zhou, S. Erfani, S. Xia, S. Wijewickrema, and J. Bailey, “Dimensionality-driven learning with noisy labels,” vol. 80, pp. 3355–3364, 10–15 Jul 2018

2018

Later among the works it cites.

S. Vascon, Y. Parin, E. Annavini, M. D’Andola, D. Zoccolan, and M. Pelillo, “Characterization of visual object representations in rat primary visual cortex,” in European Conference on Computer Vision

2018

Later among the works it cites.

A. Achille and S. Soatto, “Emergence of invariance and disentanglement in deep representations,” The Journal of Machine Learning Research

2018

Later among the works it cites.

C. Stringer, M. Pachitariu, N. Steinmetz, M. Carandini, and K. D. Harris, “High-dimensional geometry of population responses in visual cortex,” bioRxiv

2018

Later among the works it cites.

S. Chung, D. D. Lee, and H. Sompolinsky, “Classification and geometry of general perceptual manifolds,” Physical Review X

2018

Later among the works it cites.

D. G. Barrett, A. S. Morcos, and J. H. Macke, “Analyzing biological and artificial neural networks: challenges with opportunities for synergy?,” Current opinion in neurobiology

2019

Closest in time.

G. Matteucci, R. B. Marotti, M. Riggi, F. B. Rosselli, and D. Zoccolan, “Nonlinear processing of shape information in rat lateral extrastriate cortex,” Journal of Neuroscience

2019

Closest in time.