Target Networks and Over-parameterization Stabilize Off-policy Bootstrapping with Function Approximation

Understand

We prove that the combination of a target network and over-parameterized linear function approximation establishes a weaker convergence condition for bootstrapped value estimation in certain cases, even with off-policy data.

• Our condition is naturally satisfied for expected updates over the entire state-action space or learning with a batch of complete trajectories from episodic Markov decision processes.
• Notably, using only a target network or an over-parameterized model does not provide such a convergence guarantee.
• Additionally, we extend our results to learning with truncated trajectories, showing that convergence is achievable for all tasks with minor modifications, akin to value truncation for the final states in trajectories.