Batch size-invariance for policy optimization

Understand

We say an algorithm is batch size-invariant if changes to the batch size can largely be compensated for by changes to other hyperparameters.

• Stochastic gradient descent is well-known to have this property at small batch sizes, via the learning rate.
• However, some policy optimization algorithms (such as PPO) do not have this property, because of how they control the size of policy updates.
• In this work we show how to make these algorithms batch size-invariant.