Convergence of Value Aggregation for Imitation Learning

Understand

Value aggregation is a general framework for solving imitation learning problems.

• Based on the idea of data aggregation, it generates a policy sequence by iteratively interleaving policy optimization and evaluation in an online learning setting.
• While the existence of a good policy in the policy sequence can be guaranteed non-asymptotically, little is known about the convergence of the sequence or the performance of the last policy.
• In this paper, we debunk the common belief that value aggregation always produces a convergent policy sequence with improving performance.