Direct Advantage Estimation

Understand

The predominant approach in reinforcement learning is to assign credit to actions based on the expected return.

• However, we show that the return may depend on the policy in a way which could lead to excessive variance in value estimation and slow down learning.
• Instead, we show that the advantage function can be interpreted as causal effects and shares similar properties with causal representations.
• Based on this insight, we propose Direct Advantage Estimation (DAE), a novel method that can model the advantage function and estimate it directly from on-policy data while simultaneously minimizing the variance of the return without requiring the (action-)value function.