Transient Non-Stationarity and Generalisation in Deep Reinforcement Learning

Understand

Non-stationarity can arise in Reinforcement Learning (RL) even in stationary environments.

• For example, most RL algorithms collect new data throughout training, using a non-stationary behaviour policy.
• Due to the transience of this non-stationarity, it is often not explicitly addressed in deep RL and a single neural network is continually updated.
• However, we find evidence that neural networks exhibit a memory effect where these transient non-stationarities can permanently impact the latent representation and adversely affect generalisation performance.