TongUI: Internet-Scale Trajectories from Multimodal Web Tutorials for Generalized GUI Agents

Understand

Building Graphical User Interface (GUI) agents is a promising research direction, which simulates human interaction with computers or mobile phones to perform diverse GUI tasks.

• However, a major challenge in developing generalized GUI agents is the lack of sufficient trajectory data across various operating systems and applications, mainly due to the high cost of manual annotations.
• In this paper, we propose the TongUI framework that builds generalized GUI agents by learning from rich multimodal web tutorials.
• Concretely, we crawl and process online GUI tutorials (such as videos and articles) into GUI agent trajectory data, through which we produce the GUI-Net dataset containing 143K trajectory data across five operating systems and more than 200 applications.