Position: Lifetime tuning is incompatible with continual reinforcement learning

Understand

In continual RL we want agents capable of never-ending learning, and yet our evaluation methodologies do not reflect this.

• The standard practice in RL is to assume unfettered access to the deployment environment for the full lifetime of the agent.
• For example, agent designers select the best performing hyperparameters in Atari by testing each for 200 million frames and then reporting results on 200 million frames.
• In this position paper, we argue and demonstrate the pitfalls of this inappropriate empirical methodology: lifetime tuning.