Choice Between Partial Trajectories: Disentangling Goals from Beliefs

Understand

As AI agents generate increasingly sophisticated behaviors, manually encoding human preferences to guide these agents becomes more challenging.

• To address this, it has been suggested that agents instead learn preferences from human choice data.
• This approach requires a model of choice behavior that the agent can use to interpret the data.
• For choices between partial trajectories of states and actions, previous models assume choice probabilities are determined by the partial return or the cumulative advantage.