AgentGuard: Repurposing Agentic Orchestrator for Safety Evaluation of Tool Orchestration

Understand

The integration of tool use into large language models (LLMs) enables agentic systems with real-world impact.

• In the meantime, unlike standalone LLMs, compromised agents can execute malicious workflows with more consequential impact, signified by their tool-use capability.
• We propose AgentGuard, a framework to autonomously discover and validate unsafe tool-use workflows, followed by generating safety constraints to confine the behaviors of agents, achieving the baseline of safety guarantee at deployment.
• AgentGuard leverages the LLM orchestrator's innate capabilities - knowledge of tool functionalities, scalable and realistic workflow generation, and tool execution privileges - to act as its own safety evaluator.