Concave Utility Reinforcement Learning with Zero-Constraint Violations

Understand

We consider the problem of tabular infinite horizon concave utility reinforcement learning (CURL) with convex constraints.

• For this, we propose a model-based learning algorithm that also achieves zero constraint violations.
• Assuming that the concave objective and the convex constraints have a solution interior to the set of feasible occupation measures, we solve a tighter optimization problem to ensure that the constraints are never violated despite the imprecise model knowledge and model stochasticity.
• We use Bellman error-based analysis for tabular infinite-horizon setups which allows analyzing stochastic policies.