Preference-based Reinforcement Learning with Finite-Time Guarantees

Understand

Preference-based Reinforcement Learning (PbRL) replaces reward values in traditional reinforcement learning by preferences to better elicit human opinion on the target objective, especially when numerical reward values are hard to design or interpret.

• Despite promising results in applications, the theoretical understanding of PbRL is still in its infancy.
• In this paper, we present the first finite-time analysis for general PbRL problems.
• We first show that a unique optimal policy may not exist if preferences over trajectories are deterministic for PbRL.