DEMI: Discriminative Estimator of Mutual Information

Understand

Estimating mutual information between continuous random variables is often intractable and extremely challenging for high-dimensional data.

• Recent progress has leveraged neural networks to optimize variational lower bounds on mutual information.
• Although showing promise for this difficult problem, the variational methods have been theoretically and empirically proven to have serious statistical limitations: 1) many methods struggle to produce accurate estimates when the underlying mutual information is either low or high; 2) the resulting estimators may suffer from high variance.
• Our approach is based on training a classifier that provides the probability that a data sample pair is drawn from the joint distribution rather than from the product of its marginal distributions.