Understanding the Limitations of Variational Mutual Information Estimators

Understand

Variational approaches based on neural networks are showing promise for estimating mutual information (MI) between high dimensional variables.

• However, they can be difficult to use in practice due to poorly understood bias/variance tradeoffs.
• We theoretically show that, under some conditions, estimators such as MINE exhibit variance that could grow exponentially with the true amount of underlying MI.
• We also empirically demonstrate that existing estimators fail to satisfy basic self-consistency properties of MI, such as data processing and additivity under independence.