Fetching the paper…

Reading the bibliography…

2024

VideoGUI: A Benchmark for GUI Automation from Instructional Videos

Lin, Kevin Qinghong, Li, Linjie, Gao, Difei et al.

Original
PDF
Post on X

Understand

Graphical User Interface (GUI) automation holds significant promise for enhancing human productivity by assisting with computer tasks.

Existing task formulations primarily focus on simple tasks that can be specified by a single, language-only instruction, such as "Insert a new slide." In this work, we introduce VideoGUI, a novel multi-modal benchmark designed to evaluate GUI assistants on visual-centric GUI tasks.
Sourced from high-quality web instructional videos, our benchmark focuses on tasks involving professional and novel software (e.g., Adobe Photoshop or Stable Diffusion WebUI) and complex activities (e.g., video editing).
VideoGUI evaluates GUI assistants through a hierarchical process, allowing for identification of the specific levels at which they may fail: (i) high-level planning: reconstruct procedural subtasks from visual conditions without language descriptions; (ii) middle-level planning: generate sequences of precise action narrations based on visual state (i.e., screenshot) and goals; (iii) atomic action execution: perform specific actions such as accurately clicking designated elements.

VideoGUI: A Benchmark for GUI Automation from Instructional Videos · Around

Built on

World of bits: An open-domain platform for web-based agents
Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang · 2017
Earlier work this paper cites.
Mapping natural language instructions to mobile ui action sequences
Original
Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge · 2020
Earlier work this paper cites.
Androidenv: A reinforcement learning platform for android
Original
Daniel Toyama, Philippe Hamel, Anita Gergely, Gheorghe Comanici, Amelia Glaese, Zafarali Ahmed, Tyler Jackson, Shibl Mourad, and Doina Precup · 2021
Earlier work this paper cites.
Actionbert: Leveraging user actions for semantic understanding of user interfaces
Zecheng He, Srinivas Sunkara, Xiaoxue Zang, Ying Xu, Lijuan Liu, Nevan Wichers, Gabriel Schubiner, Ruby Lee, and Jindong Chen · 2021
Earlier work this paper cites.
Uibert: Learning generic multimodal representations for ui understanding
Original
Chongyang Bai, Xiaoxue Zang, Ying Xu, Srinivas Sunkara, Abhinav Rastogi, Jindong Chen, et al · 2021
Earlier work this paper cites.
Introducing chatgpt
OpenAI · 2021
Earlier work this paper cites.
Webshop: Towards scalable real-world web interaction with grounded language agents
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan · 2022
Earlier work this paper cites.
Chain-of-thought prompting elicits reasoning in large language models
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al · 2022
Earlier work this paper cites.
Gpt-4 technical report, 2023
OpenAI · 2023
Earlier work this paper cites.
A real-world webagent with planning, long context understanding, and program synthesis
Original
Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust · 2023
Earlier work this paper cites.
Webarena: A realistic web environment for building autonomous agents
Original
Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, et al · 2023
Earlier work this paper cites.
Multimodal web navigation with instruction-finetuned foundation models
Original
Hiroki Furuta, Ofir Nachum, Kuang-Huei Lee, Yutaka Matsuo, Shixiang Shane Gu, and Izzeddin Gur · 2023
Earlier work this paper cites.
Android in the wild: A large-scale dataset for android device control
Original
Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap · 2023
Earlier work this paper cites.
Empowering llm to use smartphone for intelligent task automation
Original
Hao Wen, Yuanchun Li, Guohong Liu, Shanhui Zhao, Tao Yu, Toby Jia-Jun Li, Shiqi Jiang, Yunhao Liu, Yaqin Zhang, and Yunxin Liu · 2023
Earlier work this paper cites.
Openagents: An open platform for language agents in the wild
Original
Tianbao Xie, Fan Zhou, Zhoujun Cheng, Peng Shi, Luoxuan Weng, Yitao Liu, Toh Jing Hua, Junning Zhao, Qian Liu, Che Liu, et al · 2023
Earlier work this paper cites.
Assistgui: Task-oriented desktop graphical user interface automation
Original
Difei Gao, Lei Ji, Zechen Bai, Mingyu Ouyang, Peiran Li, Dongxing Mao, Qinchen Wu, Weichen Zhang, Peiyi Wang, Xiangwu Guo, et al · 2023
Earlier work this paper cites.

Similar

Autonomous large language model agents enabling intent-driven mobile gui testing
Original
Juyeon Yoon, Robert Feldt, and Shin Yoo · 2023
Cited alongside, same era.
React: Synergizing reasoning and acting in language models
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao · 2023
Cited alongside, same era.
Mm-react: Prompting chatgpt for multimodal reasoning and action
Original
Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang · 2023
Cited alongside, same era.
Assistgpt: A general multi-modal assistant that can plan, execute, inspect, and learn
Original
Difei Gao, Lei Ji, Luowei Zhou, Kevin Qinghong Lin, Joya Chen, Zihan Fan, and Mike Zheng Shou · 2023
Cited alongside, same era.
Lexi: Self-supervised learning of the ui language

Then

Comprehensive cognitive llm agent for smartphone gui automation
Original
Xinbei Ma, Zhuosheng Zhang, and Hai Zhao · 2024
Closest in time.
Agentstudio: A toolkit for building general virtual agents
Original
Longtao Zheng, Zhiyuan Huang, Zhenghai Xue, Xinrun Wang, Bo An, and Shuicheng Yan · 2024
Closest in time.
From pixels to ui actions: Learning to follow instructions via graphical user interfaces
Peter Shaw, Mandar Joshi, James Cohan, Jonathan Berant, Panupong Pasupat, Hexiang Hu, Urvashi Khandelwal, Kenton Lee, and Kristina N Toutanova · 2024
Closest in time.
Seeclick: Harnessing gui grounding for advanced visual gui agents
Original
Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu · 2024
Closest in time.
Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Pratyay Banerjee, Shweti Mahajan, Kushal Arora, Chitta Baral, and Oriana Riva · 2023

Cited alongside, same era.

Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v, 2023

Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao · 2023

Cited alongside, same era.

Pix2struct: Screenshot parsing as pretraining for visual language understanding

Kenton Lee, Mandar Joshi, Iulia Raluca Turc, Hexiang Hu, Fangyu Liu, Julian Martin Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, and Kristina Toutanova · 2023

Cited alongside, same era.

Cogagent: A visual language model for gui agents

Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al · 2023

Cited alongside, same era.

Gpt-4 technical report

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al · 2023

Cited alongside, same era.

Mm-vet: Evaluating large multimodal models for integrated capabilities

Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang · 2023

Cited alongside, same era.

Gemini: a family of highly capable multimodal models

Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al · 2023

Cited alongside, same era.

Qwen technical report

Original

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al · 2023

Cited alongside, same era.

Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al · 2024

Closest in time.

Visualwebarena: Evaluating multimodal agents on realistic visual web tasks

Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried · 2024

Closest in time.

Screenai: A vision-language model for ui and infographics understanding

Gilles Baechler, Srinivas Sunkara, Maria Wang, Fedir Zubach, Hassan Mansoor, Vincent Etter, Victor Cărbune, Jason Lin, Jindong Chen, and Abhanshu Sharma · 2024

Closest in time.

Tie: Revolutionizing text-based image editing for complex-prompt following and high-fidelity editing

Xinyu Zhang, Mengxue Kang, Fei Wei, Shuang Xu, Yuhe Liu, and Lin Ma · 2024

Closest in time.

Ferret-ui: Grounded mobile ui understanding with multimodal llms

Original

Keen You, Haotian Zhang, Eldon Schoop, Floris Weers, Amanda Swearngin, Jeffrey Nichols, Yinfei Yang, and Zhe Gan · 2024

Closest in time.

Pyautogui

PyAutoGUI · 2024

Closest in time.

The claude 3 model family: Opus, sonnet, haiku

AI Anthropic · 2024

Closest in time.

Introducing meta llama 3: The most capable openly available llm to date, 2024

Meta · 2024

Closest in time.

Mixtral of experts

Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al · 2024

Closest in time.

Azure ocr

Azure OCR · 2024

Closest in time.

Obs studio

OBS Studio · 2024

Closest in time.