AXIS: Efficient Human-Agent-Computer Interaction with API-First LLM-Based Agents

Understand

Multimodal large language models (MLLMs) have enabled LLM-based agents to directly interact with application user interfaces (UIs), enhancing agents' performance in complex tasks.

• However, these agents often suffer from high latency and low reliability due to the extensive sequential UI interactions.
• To address this issue, we propose AXIS, a novel LLM-based agents framework that prioritize actions through application programming interfaces (APIs) over UI actions.
• This framework also facilitates the creation and expansion of APIs through automated exploration of applications.