2020

Universally Quantized Neural Compression

Agustsson, Eirikur, Theis, Lucas

Understand

A popular approach to learning encoders for lossy compression is to use additive uniform noise during training as a differentiable approximation to test-time quantization.

We demonstrate that a uniform noise channel can also be implemented at test time using universal quantization (Ziv, 1985).
This allows us to eliminate the mismatch between training and test phases while maintaining a completely differentiable loss function.
Implementing the uniform noise channel is a special case of the more general problem of communicating a sample, which we prove is computationally hard if we do not make assumptions about its distribution.

Built on

Picture Coding Using Pseudo-Random Noise
L. G. Roberts · 1962
Earlier work this paper cites.
Dither signals and their effect on quantization noise
L. Schuchman · 1964
Earlier work this paper cites.
Beyond regression: New tools for prediction and analysis in the behavioral sciences
P. Werbos · 1974
Earlier work this paper cites.
On universal quantization
J. Ziv · 1985
Earlier work this paper cites.
Information processing in dynamical systems: Foundations of harmony theory
P. Smolensky · 1987
Earlier work this paper cites.
Recommendation ITU-T T.81: Information technology – Digital compression and coding of continuous-tone still images – Requirements and guidelines, 1992
ITU-T · 1992
Earlier work this paper cites.
On universal quantization by randomized uniform/lattice quantizers
R. Zamir and M. Feder · 1992
Earlier work this paper cites.
PhotoCD PCD0992, 1993
Kodak · 1993
Earlier work this paper cites.
Entanglement-Assisted Capacity of a Quantum Channel and the Reverse Shannon Theorem
C. H. Bennett and P. W. Shor · 2002
Earlier work this paper cites.
Uniform Scale Mixture Models With Applications to Bayesian Inference
Z. Qin, P. Damien, and S. Walker · 2003
Earlier work this paper cites.
Capacity of Coordinated Actions
T. M. Cover and H. Permuter · 2007
Earlier work this paper cites.
The Communication Complexity of Correlation
P. Harsha, R. Jain, D. McAllester, and J. Radhakrishnan · 2007
Earlier work this paper cites.
Communication requirements for generating correlated random variables
P. Cuff · 2008
Earlier work this paper cites.

Similar

Restricted Boltzmann Machines are Hard to Approximately Evaluate or Simulate
P. M. Long and R. A. Servedio · 2010
Cited alongside, same era.
Estimating or propagating gradients through stochastic neurons for conditional computation, 2013
Original
Y. Bengio, N. Leonard, and A. Courville · 2013
Cited alongside, same era.
The likelihood encoder for source coding
P. W. Cuff and E. C. Song · 2013
Cited alongside, same era.
Auto-encoding variational Bayes
D. Kingma and M. Welling · 2014
Cited alongside, same era.
Adam: A Method for Stochastic Optimization
D. P. Kingma and J. Ba · 2014
Cited alongside, same era.
Stochastic backpropagation and approximate inference in deep generative models
D. J. Rezende, S. Mohamed, and D. Wierstra · 2014
Cited alongside, same era.
Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
A. M. Saxe, J. L. McClelland, and S. Ganguli · 2014
Cited alongside, same era.
Lattice Coding for Signals and Networks
R. Zamir · 2014
Cited alongside, same era.
End-to-end optimization of nonlinear transform codes for perceptual quality
J. Ballé, V. Laparra, and E. P. Simoncelli · 2016
Cited alongside, same era.
Variable rate image compression with recurrent neural networks
G. Toderici, S. M. O’Malley, S. J. Hwang, D. Vincent, D. Minnen, S. Baluja, M. Covell, and R. Sukthankar · 2016
Cited alongside, same era.
Soft-to-Hard Vector Quantization for End-to-End Learning Compressible Representations
E. Agustsson, F. Mentzer, M. Tschannen, L. Cavigelli, R. Timofte, L. Benini, and L. V. Gool · 2017
Cited alongside, same era.
State of the internet
Akamai · 2017
Cited alongside, same era.

Then

End-to-end Optimized Image Compression
J. Ballé, V. Laparra, and E. P. Simoncelli · 2017
Later among the works it cites.
Strong Functional Representation Lemma and Applications to Coding Theorems
C. T. Li and A. E. Gamal · 2017
Later among the works it cites.
Real-time adaptive image compression
O. Rippel and L. Bourdev · 2017
Later among the works it cites.
Lossy Image Compression with Compressive Autoencoders
L. Theis, W. Shi, A. Cunningham, and F. Huszár · 2017
Later among the works it cites.
Variational Image Compression with a Scale Hyperprior
Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston · 2018
Later among the works it cites.
Cisco Visual Networking Index (VNI), 2018
T. Barnett Jr., S. Jain, U. Andra, and T. Khurana · 2018
Later among the works it cites.
Joint Autoregressive and Hierarchical Priors for Learned Image Compression
D. Minnen, J. Ballé, and G. D. Toderici · 2018
Later among the works it cites.
Variational Autoencoder for Low Bit-rate Image Compression
L. Zhou, C. Cai, Y. Gao, S. Su, and J. Wu · 2018
Later among the works it cites.
Generative adversarial networks for extreme learned image compression
E. Agustsson, M. Tschannen, F. Mentzer, R. Timofte, and L. Van Gool · 2019
Later among the works it cites.
Variable rate deep image compression with a conditional autoencoder
Y. Choi, M. El-Khamy, and J. Lee · 2019
Later among the works it cites.
Minimal Random Code Learning: Getting Bits Back from Compressed Model Parameters
M. Havasi, R. Peharz, and J. M. Hernández-Lobato · 2019
Later among the works it cites.
Compressing Images by Encoding Their Latent Representations with Relative Entropy Coding, 2020
G. Flamich, M. Havasi, and J. M. Hernández-Lobato · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…