Fetching the paper…
Reading the bibliography…
We study the flow of information and the evolution of internal representations during deep neural network (DNN) training, aiming to demystify the compression aspect of the information bottleneck theory.
Asymptotically efficient estimation of nonlinear functionals
Levit, B. Y · 1978
Earlier work this paper cites.
Limit theorems for sums of general functions of m-spacings
Hall, P · 1984
Earlier work this paper cites.
Sample estimate of the entropy of a random vector
Kozachenko, L. F. and Leonenko, N. N · 1987
Earlier work this paper cites.
Self-organization in a perceptual network
Linsker, R · 1988
Earlier work this paper cites.
Estimation of entropy and other functionals of a multivariate density
Joe, H · 1989
Earlier work this paper cites.
On the estimation of entropy
Hall, P. and Morton, S. C · 1993
Earlier work this paper cites.
Root- n n consistent estimators of entropy for densities with unbounded support
Tsybakov, A. B. and Van der Meulen, E. C · 1996
Earlier work this paper cites.
A general lower bound of minimax risk for absolute-error loss
Chen, J · 1997
Earlier work this paper cites.
Gradient-based learning applied to document recognition
LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P · 1999
Earlier work this paper cites.
Estimation of entropy and mutual information
Paninski, L · 2003
Earlier work this paper cites.
Estimating mutual information
Kraskov, A., Stögbauer, H., and Grassberger, P · 2004
Earlier work this paper cites.
Monte Carlo Methods
Robert, C. P · 2004
Earlier work this paper cites.
Elements of Information Theory
Cover, T. M. and Thomas, J. A · 2006
Cited alongside, same era.
Optimal transport: old and new , volume 338
Villani, C · 2006
Cited alongside, same era.
On entropy estimation by m-spacing method
Haje, H. F. E. and Golubev, Y · 2009
Cited alongside, same era.
A tail inequality for quadratic forms of subgaussian random vectors
Hsu, D., Kakade, S., and Zhang, T · 2012
Cited alongside, same era.
Estimation of nonlinear functionals of densities with confidence
Sricharan, K., Raich, R., and Hero, A. O · 2012
Cited alongside, same era.
Nonparametric von Mises estimators for entropies, divergences and mutual informations
Kandasamy, K., Krishnamurthy, A., Poczos, B., Wasserman, L., and Robins, J. M · 2015
Cited alongside, same era.
Automatic differentiation in PyTorch
Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., and Lerer, A · 2017
Later among the works it cites.
Lecture notes on information theory
Polyanskiy, Y. and Wu, Y · 2017
Later among the works it cites.
Opening the black box of deep neural networks via information
Shwartz-Ziv, R. and Tishby, N · 2017
Later among the works it cites.
On the emergence of invariance and disentangling in deep representations
Achille, A. and Soatto, S · 2018
Closest in time.
Amjad, R. A. and Geiger, B. C · 2018
Closest in time.
Mutual information neural estimation
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Wasserstein continuity of entropy and outer bounds for interference channels
Polyanskiy, Y. and Wu, Y · 2016
Cited alongside, same era.
Finite-sample analysis of fixed-k nearest neighbor density functional estimators
Singh, S. and Póczos, B · 2016
Cited alongside, same era.
Minimax rates of entropy estimation on large alphabets via best polynomial approximation
Wu, Y. and Yang, P · 2016
Cited alongside, same era.
Parseval networks: Improving robustness to adversarial examples
Cisse, M., Bojanowski, P., Grave, E., Dauphin, Y., and Usunier, N · 2017
Cited alongside, same era.
Optimal rates of entropy estimation over Lipschitz balls
Han, Y., Jiao, J., Weissman, T., and Wu, Y · 2017
Cited alongside, same era.
Estimating mixture entropy with pairwise distances
Kolchinsky, A. and Tracey, B. D · 2017
Cited alongside, same era.
Belghazi, M. I., Baratin, A., Rajeswar, S., Ozair, S., Bengio, Y., Courville, A., and Hjelm, R. D · 2018
Closest in time.
On the information bottleneck theory of deep learning
Saxe, A. M., Bansal, Y., Dapello, J., Advani, M., Kolchinsky, A., Tracey, B. D., and Cox, D. D · 2018
Closest in time.
Representation learning with contrastive predictive coding
van den Oord, A., Li, Y., and Vinyals, O · 2018
Closest in time.
Efficient multivariate entropy estimation via k k -nearest neighbour distances
Berrett, T. B., Samworth, R. J., and Yuan, M · 2019
Closest in time.
Optimality of the plug-in estimator for differential entropy estimation under Gaussian convolutions
Goldfeld, Z., Greenewald, K., Weed, J., and Polyanskiy, Y · 2019
Closest in time.
Learning deep representations by mutual information estimation and maximization
Hjelm, R. D., Fedorov, A., Lavoie-Marchildon, S., Grewal, K., Bachman, P., Trischler, A., and Bengio, Y · 2019
Closest in time.
Caveats for information bottleneck in deterministic scenarios
Kolchinsky, A., Tracey, B. D., and Van Kuyk, S · 2019
Closest in time.