On the Estimation of Information Measures of Continuous Distributions

Understand

The estimation of information measures of continuous distributions based on samples is a fundamental problem in statistics and machine learning.

• In this paper, we analyze estimates of differential entropy in $K$-dimensional Euclidean space, computed from a finite number of samples, when the probability density function belongs to a predetermined convex family $\mathcal{P}$.
• First, estimating differential entropy to any accuracy is shown to be infeasible if the differential entropy of densities in $\mathcal{P}$ is unbounded, clearly showing the necessity of additional assumptions.
• Subsequently, we investigate sufficient conditions that enable confidence bounds for the estimation of differential entropy.