Unifying Gradient Estimators for Meta-Reinforcement Learning via Off-Policy Evaluation

Understand

Model-agnostic meta-reinforcement learning requires estimating the Hessian matrix of value functions.

• This is challenging from an implementation perspective, as repeatedly differentiating policy gradient estimates may lead to biased Hessian estimates.
• In this work, we provide a unifying framework for estimating higher-order derivatives of value functions, based on off-policy evaluation.
• Our framework interprets a number of prior approaches as special cases and elucidates the bias and variance trade-off of Hessian estimates.