Fetching the paper…

Reading the bibliography…

2018

Variational Information Bottleneck on Vector Quantized Autoencoders

Wu, Hanwei, Flierl, Markus

Original
PDF
Post on X

Understand

In this paper, we provide an information-theoretic interpretation of the Vector Quantized-Variational Autoencoder (VQ-VAE).

We show that the loss function of the original VQ-VAE can be derived from the variational deterministic information bottleneck (VDIB) principle.
On the other hand, the VQ-VAE trained by the Expectation Maximization (EM) algorithm can be viewed as an approximation to the variational information bottleneck(VIB) principle.

Variational Information Bottleneck on Vector Quantized Autoencoders · Around

Built on

“An information theoretic tradeoff between complexity and accuracy,”
A. Gilad-Bachrach, A. Navot, and N. Tishby, · 2003
Earlier work this paper cites.
“Auto-encoding variational bayes,”
D. P. Kingma and M. Welling, · 2014
Earlier work this paper cites.
“Neural discrete representation learning,”
A. Oord, K. Kavukcuoglu, and O. Vinyals, · 2017
Earlier work this paper cites.

Similar

“Deep variational information bottleneck,”
A.A. Alemi, I. Fischer, J. V. Dillon, and K. Murphy, · 2017
Cited alongside, same era.
“The deterministic information bottleneck,”
DJ Strouse and D. Schwab, · 2017
Cited alongside, same era.
“The information bottleneck method,”
N. Tishby, F. C. Pereira, and W. Bialek,
Cited in the paper.
“Deep learning and the information bottleneck principle,”
N. Tishby and N. Zaslavsky,
Cited in the paper.

Then

“Variational deterministic information bottleneck,” 2018,
DJ Strouse and D. Schwab, · 2018
Closest in time.
“Theory and experiments on vector quantized autoencoders,” 2018,
Original
A. Roy, A. Vaswani, A. Neelakantan, and N. Parmar, · 2018
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…