From Feature Importance to Natural Language Explanations Using LLMs with RAG

Understand

As machine learning becomes increasingly integral to autonomous decision-making processes involving human interaction, the necessity of comprehending the model's outputs through conversational means increases.

• Most recently, foundation models are being explored for their potential as post hoc explainers, providing a pathway to elucidate the decision-making mechanisms of predictive models.
• In this work, we introduce traceable question-answering, leveraging an external knowledge repository to inform the responses of Large Language Models (LLMs) to user queries within a scene understanding task.
• This knowledge repository comprises contextual details regarding the model's output, containing high-level features, feature importance, and alternative probabilities.