2020

CLUB: A Contrastive Log-ratio Upper Bound of Mutual Information

Cheng, Pengyu, Hao, Weituo, Dai, Shuyang et al.

Understand

Mutual information (MI) minimization has gained considerable interests in various machine learning tasks.

However, estimating and minimizing MI in high-dimensional spaces remains a challenging problem, especially when only samples, rather than distribution forms, are accessible.
Previous works mainly focus on MI lower bound approximation, which is not applicable to MI minimization problems.
In this paper, we propose a novel Contrastive Log-ratio Upper Bound (CLUB) of mutual information.

Built on

Using the mutual information coefficient to identify lags in nonlinear models
Granger, C. and Lin, J.-L · 1994
Earlier work this paper cites.
Information theory and statistics
Kullback, S · 1997
Earlier work this paper cites.
Estimation of the information by an adaptive partitioning of the observation space
Darbellay, G. A. and Vajda, I · 1999
Earlier work this paper cites.
Neural networks for density estimation
Magdon-Ismail, M. and Atiya, A. F · 1999
Earlier work this paper cites.
The information bottleneck method
Tishby, N., Pereira, F. C., and Bialek, W · 2000
Earlier work this paper cites.
The im algorithm: a variational approach to information maximization
Barber, D. and Agakov, F. V · 2003
Earlier work this paper cites.
Nonparametric and Semiparametric Models
Härdle, W. K., Müller, M., Sperlich, S., and Werwatz, A · 2004
Earlier work this paper cites.
Estimating mutual information
Kraskov, A., Stögbauer, H., and Grassberger, P · 2004
Earlier work this paper cites.
Approximating mutual information by maximum likelihood density ratio estimation
Suzuki, T., Sugiyama, M., Sese, J., and Kanamori, T · 2008
Earlier work this paper cites.
Noise-contrastive estimation: A new estimation principle for unnormalized statistical models
Gutmann, M. and Hyvärinen, A · 2010
Earlier work this paper cites.
Estimating divergence functionals and the likelihood ratio by convex risk minimization
Nguyen, X., Wainwright, M. J., and Jordan, M. I · 2010
Earlier work this paper cites.
Fairness-aware learning through regularization approach
Kamishima, T., Akaho, S., and Sakuma, J · 2011
Earlier work this paper cites.
Improving neural networks by preventing co-adaptation of feature detectors
Original
Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. R · 2012
Earlier work this paper cites.
Auto-encoding variational bayes
Original
Kingma, D. P. and Welling, M · 2013
Earlier work this paper cites.
Distributed representations of words and phrases and their compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., and Dean, J · 2013
Earlier work this paper cites.

Similar

On mutual information-based control of range sensing robots for mapping applications
Julian, B. J., Karaman, S., and Rus, D · 2014
Cited alongside, same era.
Information-theoretic mapping using cauchy-schwarz quadratic mutual information
Charrow, B., Liu, S., Kumar, V., and Michael, N · 2015
Cited alongside, same era.
Nonparametric k-sample tests via dynamic slicing
Jiang, B., Ye, C., and Liu, J. S · 2015
Cited alongside, same era.
Deep variational information bottleneck
Original
Alemi, A. A., Fischer, I., Dillon, J. V., and Murphy, K · 2016
Cited alongside, same era.
Domain separation networks
Bousmalis, K., Trigeorgis, G., Silberman, N., Krishnan, D., and Erhan, D · 2016
Cited alongside, same era.
Infogan: Interpretable representation learning by information maximizing generative adversarial nets
Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., and Abbeel, P · 2016
Cited alongside, same era.
Domain-adversarial training of neural networks
Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., and Lempitsky, V · 2016
Cited alongside, same era.
node2vec: Scalable feature learning for networks
Grover, A. and Leskovec, J · 2016
Cited alongside, same era.
Aracne-ap: gene network reverse engineering through adaptive partitioning inference of mutual information
Lachmann, A., Giorgi, F. M., Lopez, G., and Califano, A · 2016
Cited alongside, same era.
Mitos. jl: mutual information tools for protein sequence analysis in the julia language
Zea, D. J., Anfossi, D., Nielsen, M., and Marino-Buslje, C · 2016
Cited alongside, same era.
Stochastic neural networks for hierarchical reinforcement learning
Original
Florensa, C., Duan, Y., and Abbeel, P · 2017
Cited alongside, same era.
Learning discrete representations via information maximizing self-augmented training
Hu, W., Miyato, T., Tokui, S., Matsumoto, E., and Sugiyama, M · 2017
Cited alongside, same era.

Then

Learning deep representations by mutual information estimation and maximization
Original
Hjelm, R. D., Fedorov, A., Lavoie-Marchildon, S., Grewal, K., Bachman, P., Trischler, A., and Bengio, Y · 2018
Later among the works it cites.
Unsupervised image-to-image translation using domain-specific variational information bound
Kazemi, H., Soleymani, S., Taherkhani, F., Iranmanesh, S., and Nasrabadi, N · 2018
Later among the works it cites.
Disentangling by factorising
Kim, H. and Mnih, A · 2018
Later among the works it cites.
Disentangled person image generation
Ma, L., Sun, Q., Georgoulis, S., Van Gool, L., Schiele, B., and Fritz, M · 2018
Later among the works it cites.
Representation learning with contrastive predictive coding
Original
Oord, A. v. d., Li, Y., and Vinyals, O · 2018
Later among the works it cites.
Maximum classifier discrepancy for unsupervised domain adaptation
Saito, K., Watanabe, K., Ushiku, Y., and Harada, T · 2018
Later among the works it cites.
Variational information distillation for knowledge transfer
Ahn, S., Hu, S. X., Damianou, A., Lawrence, N. D., and Dai, Z · 2019
Later among the works it cites.
Contrastively smoothed class alignment for unsupervised domain adaptation
Original
Dai, S., Cheng, Y., Zhang, Y., Gan, Z., Liu, J., and Carin, L · 2019
Later among the works it cites.
Understanding generalization of deep neural networks trained with noisy labels
Original
Hu, W., Li, Z., and Yu, D · 2019
Later among the works it cites.
Disentangled representation learning for non-parallel text style transfer
John, V., Mou, L., Bahuleyan, H., and Vechtomova, O · 2019
Later among the works it cites.
Challenging common assumptions in the unsupervised learning of disentangled representations
Locatello, F., Bauer, S., Lucic, M., Raetsch, G., Gelly, S., Schölkopf, B., and Bachem, O · 2019
Later among the works it cites.
Overparameterized nonlinear learning: Gradient descent takes the shortest path?
Oymak, S. and Soltanolkotabi, M · 2019
Later among the works it cites.
On variational bounds of mutual information
Poole, B., Ozair, S., Van Den Oord, A., Alemi, A., and Tucker, G · 2019
Later among the works it cites.
Improving disentangled text representation learning with information-theoretic guidance
Original
Cheng, P., Min, M. R., Shen, D., Malon, C., Zhang, Y., Li, Y., and Carin, L · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…