A unified view of likelihood ratio and reparameterization gradients and an optimal importance sampling scheme

Understand

Reparameterization (RP) and likelihood ratio (LR) gradient estimators are used throughout machine and reinforcement learning; however, they are usually explained as simple mathematical tricks without providing any insight into their nature.

• We use a first principles approach to explain LR and RP, and show a connection between the two via the divergence theorem.
• The theory motivated us to derive optimal importance sampling schemes to reduce LR gradient variance.
• Our newly derived distributions have analytic probability densities and can be directly sampled from.