AlignIQL: Policy Alignment in Implicit Q-Learning through Constrained Optimization

Understand

Implicit Q-learning (IQL) serves as a strong baseline for offline RL, which learns the value function using only dataset actions through quantile regression.

• However, it is unclear how to recover the implicit policy from the learned implicit Q-function and why IQL can utilize weighted regression for policy extraction.
• IDQL reinterprets IQL as an actor-critic method and gets weights of implicit policy, however, this weight only holds for the optimal value function.
• In this work, we introduce a different way to solve the implicit policy-finding problem (IPF) by formulating this problem as an optimization problem.