Interestingness Elements for Explainable Reinforcement Learning: Understanding Agents' Capabilities and Limitations

Understand

We propose an explainable reinforcement learning (XRL) framework that analyzes an agent's history of interaction with the environment to extract interestingness elements that help explain its behavior.

• The framework relies on data readily available from standard RL algorithms, augmented with data that can easily be collected by the agent while learning.
• We describe how to create visual summaries of an agent's behavior in the form of short video-clips highlighting key interaction moments, based on the proposed elements.
• We also report on a user study where we evaluated the ability of humans to correctly perceive the aptitude of agents with different characteristics, including their capabilities and limitations, given visual summaries automatically generated by our framework.