Infinite Shannon entropy

Understand

Even if a probability distribution is properly normalizable, its associated Shannon (or von Neumann) entropy can easily be infinite.

• We carefully analyze conditions under which this phenomenon can occur.
• Roughly speaking, this happens when arbitrarily small amounts of probability are dispersed into an infinite number of states; we shall quantify this observation and make it precise.
• We develop several particularly simple, elementary, and useful bounds, and also provide some asymptotic estimates, leading to necessary and sufficient conditions for the occurrence of infinite Shannon entropy.