On the Properties of Kullback-Leibler Divergence Between Multivariate Gaussian Distributions

Understand

Kullback-Leibler (KL) divergence is one of the most important divergence measures between probability distributions.

• In this paper, we prove several properties of KL divergence between multivariate Gaussian distributions.
• First, for any two $n$-dimensional Gaussian distributions $\mathcal{N}_1$ and $\mathcal{N}_2$, we give the supremum of $KL(\mathcal{N}_1||\mathcal{N}_2)$ when $KL(\mathcal{N}_2||\mathcal{N}_1)\leq \varepsilon\ (\varepsilon>0)$.
• For small $\varepsilon$, we show that the supremum is $\varepsilon + 2\varepsilon^{1.5} + O(\varepsilon^2)$.