Fetching the paper…

Reading the bibliography…

2014

Notes on Kullback-Leibler Divergence and Likelihood

Shlens, Jonathon

Original
PDF
Post on X

Understand

The Kullback-Leibler (KL) divergence is a fundamental equation of information theory that quantifies the proximity of two probability distributions.

Although difficult to understand by examining the equation, an intuition and understanding of the KL divergence arises from its intimate relationship with likelihood theory.
We discuss how KL divergence arises from likelihood theory in an attempt to provide some intuition and reserve a rigorous (but rather simple) derivation for the appendix.
Finally, we comment on recent applications of KL divergence in the neural coding literature and highlight its natural application.

Notes on Kullback-Leibler Divergence and Likelihood · Around

Built on

The mathematical theory of communication
CE Shannon and W Weaver · 1949
Earlier work this paper cites.
Elements of information theory
TM Cover and JA Thomas · 1991
Earlier work this paper cites.
Pattern classification
RO Duda, PE Hart, and DG Stork · 2001
Earlier work this paper cites.

Similar

Probability Theory: The Logic of Science
ET Jaynes · 2003
Cited alongside, same era.
Network information and connected correlations
E Schneidman, S Still, MJ Berry, and W Bialek · 2003
Cited alongside, same era.
Synergy, redundancy, and independence in population codes, revisited
PE Latham and S Nirenberg · 2005
Cited alongside, same era.

Then

Correlation and independence in the neural code
S Amari and H Nakahara · 2006
Later among the works it cites.
Weak pairwise correlations imply strongly correlated network states in a neural population
E Schneidman, MJ Berry, R Segev, and W Bialek · 2006
Later among the works it cites.
The structure of multi-neuron firing patterns in primate retina
Original
J Shlens, GD Field, JL Gauthier, MI Grivich, D Petrusca, A Sher, AM Litke, and EJ Chichilnisky · 2006
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…