A Natural Actor-Critic Algorithm with Downside Risk Constraints

Understand

Existing work on risk-sensitive reinforcement learning - both for symmetric and downside risk measures - has typically used direct Monte-Carlo estimation of policy gradients.

• While this approach yields unbiased gradient estimates, it also suffers from high variance and decreased sample efficiency compared to temporal-difference methods.
• In this paper, we study prediction and control with aversion to downside risk which we gauge by the lower partial moment of the return.
• We introduce a new Bellman equation that upper bounds the lower partial moment, circumventing its non-linearity.