AppAgent: Multimodal Agents as Smartphone Users

Understand

Recent advancements in large language models (LLMs) have led to the creation of intelligent agents capable of performing complex tasks.

• This paper introduces a novel LLM-based multimodal agent framework designed to operate smartphone applications.
• Our framework enables the agent to operate smartphone applications through a simplified action space, mimicking human-like interactions such as tapping and swiping.
• This novel approach bypasses the need for system back-end access, thereby broadening its applicability across diverse apps.