2019

Information in Infinite Ensembles of Infinitely-Wide Neural Networks

Shwartz-Ziv, Ravid, Alemi, Alexander A.

Understand

In this preliminary work, we study the generalization properties of infinite ensembles of infinitely-wide neural networks.

Amazingly, this model family admits tractable calculations for many information-theoretic quantities.
We report analytical and empirical investigations in the search for signals that correlate with generalization.

On variational bounds of mutual information
Original
Ben Poole, Sherjil Ozair, Aäron van den Oord, Alexander A. Alemi, and George Tucker · 1905
Earlier work this paper cites.
Information bottleneck for gaussian variables
Gal Chechik, Amir Globerson, Naftali Tishby, and Yair Weiss · 2005
Earlier work this paper cites.
On bayesian bounds
Arindam Banerjee · 2006
Earlier work this paper cites.
MNIST handwritten digit database
Yann LeCun and Corinna Cortes · 2010
Earlier work this paper cites.
Elements of information theory
Thomas M Cover and Joy A Thomas · 2012
Earlier work this paper cites.
Deep variational information bottleneck
Original
Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy · 2016
Earlier work this paper cites.
Emergence of Invariance and Disentangling in Deep Representations
A. Achille and S. Soatto · 2017
Earlier work this paper cites.

Learners that use little information, 2017
Raef Bassily, Shay Moran, Ido Nachum, Jonathan Shafer, and Amir Yehudayoff · 2017
Cited alongside, same era.
Opening the black box of deep neural networks via information
Original
Ravid Shwartz-Ziv and Naftali Tishby · 2017
Cited alongside, same era.
How (not) to train your neural network using the information bottleneck principle
Original
Rana Ali Amjad and Bernhard C Geiger · 2018
Cited alongside, same era.
Neural ordinary differential equations, 2018
Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud · 2018
Cited alongside, same era.
Neural tangent kernel: Convergence and generalization in neural networks
Arthur Jacot, Franck Gabriel, and Clément Hongler · 2018
Cited alongside, same era.
Caveats for information bottleneck in deterministic scenarios
Original
Artemy Kolchinsky, Brendan D Tracey, and Steven Van Kuyk · 2018
Cited alongside, same era.
On the information bottleneck theory of deep learning
Andrew Michael Saxe, Yamini Bansal, Joel Dapello, Madhu Advani, Artemy Kolchinsky, Brendan Daniel Tracey, and David Daniel Cox · 2018
Cited alongside, same era.

Where is the information in a deep neural network?, 2019
Alessandro Achille and Stefano Soatto · 2019
Closest in time.
Harnessing the power of infinitely wide deep nets on small-data tasks, 2019
Sanjeev Arora, Simon S. Du, Zhiyuan Li, Ruslan Salakhutdinov, Ruosong Wang, and Dingli Yu · 2019
Closest in time.
Limitations of the empirical fisher approximation, 2019
Frederik Kunstner, Lukas Balles, and Philipp Hennig · 2019
Closest in time.
Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent
Original
Jaehoon Lee, Lechao Xiao, Samuel S. Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, and Jeffrey Pennington · 2019
Closest in time.
Deep learning and the information bottleneck principle
Naftali Tishby and Noga Zaslavsky · 2019
Closest in time.
Neural tangents: Fast and easy infinite neural networks in python
Roman Novak, Lechao Xiao, Jiri Hron, Jaehoon Lee, Alexander A. Alemi, Jascha Sohl-Dickstein, and Samuel S. Schoenholz · 2020
Closest in time.

alphaXiv searches the wider corpus for related work and actual follow-ups.

alphaXiv is searching for related work…

Reading the bibliography…