MuProp: Unbiased Backpropagation for Stochastic Neural Networks

Understand

Deep neural networks are powerful parametric models that can be trained efficiently using the backpropagation algorithm.

• Stochastic neural networks combine the power of large parametric functions with that of graphical models, which makes it possible to learn very complex distributions.
• However, as backpropagation is not directly applicable to stochastic networks that include discrete sampling operations within their computational graph, training such networks remains difficult.
• We present MuProp, an unbiased gradient estimator for stochastic networks, designed to make this task easier.