Optimal Kronecker-Sum Approximation of Real Time Recurrent Learning

Understand

One of the central goals of Recurrent Neural Networks (RNNs) is to learn long-term dependencies in sequential data.

• Nevertheless, the most popular training method, Truncated Backpropagation through Time (TBPTT), categorically forbids learning dependencies beyond the truncation horizon.
• In contrast, the online training algorithm Real Time Recurrent Learning (RTRL) provides untruncated gradients, with the disadvantage of impractically large computational costs.
• Recently published approaches reduce these costs by providing noisy approximations of RTRL.