Fetching the paper…

Reading the bibliography…

2016

Flow of Information in Feed-Forward Deep Neural Networks

Khadivi, Pejman, Tandon, Ravi, Ramakrishnan, Naren

Original
PDF
Post on X

Understand

Feed-forward deep neural networks have been used extensively in various machine learning applications.

Developing a precise understanding of the underling behavior of neural networks is crucial for their efficient deployment.
In this paper, we use an information theoretic approach to study the flow of information in a neural network and to determine how entropy of information changes between consecutive layers.
Moreover, using the Information Bottleneck principle, we develop a constrained optimization problem that can be used in the training process of a deep neural network.

Flow of Information in Feed-Forward Deep Neural Networks · Around

Built on

“Computation of channel capacity and rate-distortion functions,”
Original
R.E. Blahut, · 1972
Earlier work this paper cites.
“Big data deep learning: Challenges and perspectives,”
X.W. Chen and X.Lin, · 1991
Earlier work this paper cites.
“Theoretical foundations of neural networks,”
C. M. Bishop, · 1996
Earlier work this paper cites.
Machine Learning
T.M. Mitchell, · 1997
Earlier work this paper cites.
“The information bottleneck method,”
N. Tishby, F. C. Pereira, and W. Bialek, · 1999
Earlier work this paper cites.

Similar

Structural, Syntactic, and Statistical Pattern Recognition: Joint IAPR International Workshops SSPR 2002 and SPR 2002 Windsor, Ontario, Canada, August 6–9, 2002 Proceedings
T.G. Dietterich, · 2002
Cited alongside, same era.
“Reducing the dimensionality of data with neural networks,”
G. E. Hinton and R. R. Salakhutdinov, · 2006
Cited alongside, same era.
“Deep neural networks for object detection,”
C. Szegedy, A. Toshev, and D. Erhan, · 2013
Cited alongside, same era.
“Deep learning face representation from predicting 10,000 classes,”
Y. Sun, X. Wang, and X. Tang, · 2014
Cited alongside, same era.

Then

“Deep learning in neural networks: An overview,”
J. Schmidhuber, · 2014
Later among the works it cites.
“An exact mapping between the Variational Renormalization Group and Deep Learning,”
P. Mehta and D. J. Schwab, · 2014
Later among the works it cites.
“Deep learning for detecting robotic grasps,”
I. Lenz, H. Lee, and A. Saxena, · 2015
Later among the works it cites.
“Deep learning and the information bottleneck principle,”
N.Tishby and N.Zaslavsky, · 2015
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…