Greedy Actor-Critic: A New Conditional Cross-Entropy Method for Policy Improvement

Understand

Many policy gradient methods are variants of Actor-Critic (AC), where a value function (critic) is learned to facilitate updating the parameterized policy (actor).

• The update to the actor involves a log-likelihood update weighted by the action-values, with the addition of entropy regularization for soft variants.
• In this work, we explore an alternative update for the actor, based on an extension of the cross entropy method (CEM) to condition on inputs (states).
• The idea is to start with a broader policy and slowly concentrate around maximal actions, using a maximum likelihood update towards actions in the top percentile per state.