ClickAgent: Enhancing UI Location Capabilities of Autonomous Agents

Webgpt: Browser-assisted question-answering with human feedback

Original

Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. 2021 · 2021

Earlier work this paper cites.

Cogagent: A visual language model for gui agents

Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. 2023 · 2023

Earlier work this paper cites.

Android in the wild: a large-scale dataset for android device control

Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. 2023 · 2023

Earlier work this paper cites.

Appagent: Multimodal agents as smartphone users

Original

Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. 2023 · 2023

Earlier work this paper cites.

Ferret: Refer and ground anything anywhere at any granularity

Original

Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. 2023 · 2023

Earlier work this paper cites.

Agenttuning: Enabling generalized agent abilities for llms

Original

Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. 2023 · 2023

Earlier work this paper cites.

You only look at screens: Multimodal chain-of-action agents

Zhuosheng Zhan and Aston Zhang. 2023 · 2023

Earlier work this paper cites.

Screenai: A vision-language model for ui and infographics understanding

Gilles Baechler, Srinivas Sunkara, Maria Wang, Fedir Zubach, Hassan Mansoor, Vincent Etter, Victor Cărbune, Jason Lin, Jindong Chen, and Abhanshu Sharma. 2024 · 2024

Earlier work this paper cites.

Guicourse: From general vision language models to versatile gui agents

Original

Wentong Chen, Junbo Cui, Jinyi Hu, Yujia Qin, Junjie Fang, Yue Zhao, Chongyi Wang, Jun Liu, Guirong Chen, Yupeng Huo, et al. 2024 · 2024

Earlier work this paper cites.

Seeclick: Harnessing gui grounding for advanced visual gui agents

Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. 2024 · 2024

Cited alongside, same era.

From explicit cot to implicit cot: Learning to internalize cot step by step

Original

Yuntian Deng, Yejin Choi, and Stuart Shieber. 2024 · 2024

Cited alongside, same era.

An interactive agent foundation model

Original

Zane Durante, Bidipta Sarkar, Ran Gong, Rohan Taori, Yusuke Noda, Paul Tang, Ehsan Adeli, Shrinidhi Kowshika Lakshmikanth, Kevin Schulman, Arnold Milstein, et al. 2024 · 2024

Cited alongside, same era.

Read anywhere pointed: Layout-aware gui screen reading with tree-of-lens grounding

Original

Yue Fan, Lei Ding, Ching-Chen Kuo, Shan Jiang, Yang Zhao, Xinze Guan, Jie Yang, Yi Zhang, and Xin Eric Wang. 2024 · 2024

Cited alongside, same era.

Enhancing vision-language pre-training with rich supervisions

Coco-agent: A comprehensive cognitive mllm agent for smartphone gui automation

Xinbei Ma, Zhuosheng Zhang, and Hai Zhao. 2024 · 2024

Closest in time.

Autonomous evaluation and refinement of digital agents

Jiayi Pan, Yichi Zhang, Nicholas Tomlin, Yifei Zhou, Sergey Levine, and Alane Suhr. 2024 · 2024

Closest in time.

Tinyclick: Single-turn agent for empowering gui automation

Original

Pawel Pawlowski, Krystian Zawistowski, Wojciech Lapacz, Marcin Skorupa, Adam Wiacek, Sebastien Postansque, and Jakub Hoscilowicz. 2024 · 2024

Closest in time.

Let’s think dot by dot: Hidden computation in transformer language models

Original

Jacob Pfau, William Merrill, and Samuel R Bowman. 2024 · 2024

Closest in time.

V-zen: Efficient gui understanding and precise grounding with a novel multimodal llm

Original

Abdur Rahman, Rajat Chawla, Muskaan Kumar, Arkajit Datta, Adarsh Jha, Mukunda NS, and Ishaan Bhola. 2024 · 2024