Recurrent Reinforcement Learning with Memoroids

Understand

Memory models such as Recurrent Neural Networks (RNNs) and Transformers address Partially Observable Markov Decision Processes (POMDPs) by mapping trajectories to latent Markov states.

• Neither model scales particularly well to long sequences, especially compared to an emerging class of memory models called Linear Recurrent Models.
• We discover that the recurrent update of these models resembles a monoid, leading us to reformulate existing models using a novel monoid-based framework that we call memoroids.
• We revisit the traditional approach to batching in recurrent reinforcement learning, highlighting theoretical and empirical deficiencies.