LPO: Towards Accurate GUI Agent Interaction via Location Preference Optimization

Understand

The advent of autonomous agents is transforming interactions with Graphical User Interfaces (GUIs) by employing natural language as a powerful intermediary.

• Despite the predominance of Supervised Fine-Tuning (SFT) methods in current GUI agents for achieving spatial localization, these methods face substantial challenges due to their limited capacity to accurately perceive positional data.
• Existing strategies, such as reinforcement learning, often fail to assess positional accuracy effectively, thereby restricting their utility.
• In response, we introduce Location Preference Optimization (LPO), a novel approach that leverages locational data to optimize interaction preferences.