Fetching the paper…
Reading the bibliography…
Recent Computer-Using Agents (CUAs), powered by multimodal large language models (LLMs), offer a promising direction for automating complex desktop workflows through natural language.
Modern languages and microsoft’s component object model
David N Gray, John Hotchkiss, Seth LaForge, Andrew Shalit, and Toby Weinberg · 1998
Earlier work this paper cites.
Virtualization: virtually at the desktop
Karissa Miller and Mahmoud Pegah · 2007
Earlier work this paper cites.
Evaluation of inter-process communication mechanisms
Aditya Venkataraman and Kishore Kumar Jagadeesha · 2015
Earlier work this paper cites.
The future digital work force: robotic process automation (rpa)
Somayya Madakam, Rajesh M Holmukhe, and Durgesh Kumar Jaiswal · 2019
Earlier work this paper cites.
Sentence-bert: Sentence embeddings using siamese bert-networks
Nils Reimers and Iryna Gurevych · 2019
Earlier work this paper cites.
Robotic process automation
Peter Hofmann, Caroline Samp, and Nils Urbach · 2020
Earlier work this paper cites.
Retrieval-augmented generation for knowledge-intensive nlp tasks
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al · 2020
Earlier work this paper cites.
Robotic process automation for industry: adoption status, benefits, challenges and research agenda
Dhanya Pramod · 2022
Earlier work this paper cites.
Chain-of-thought prompting elicits reasoning in large language models
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al · 2022
Earlier work this paper cites.
Rethinking the role of demonstrations: What makes in-context learning work?
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer · 2022
Earlier work this paper cites.
A survey of large language models
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al · 2023
Earlier work this paper cites.
Towards intelligent automation (ia): literature review on the evolution of robotic process automation (rpa), its challenges, and future trends
Julia Siderska, Lili Aunimo, Thomas Süße, John von Stamm, Damian Kedziora, and Suraya Nabilah Binti Mohd Aini · 2023
Earlier work this paper cites.
The dawn of lmms: Preliminary explorations with gpt-4v (ision)
Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang · 2023
Earlier work this paper cites.
React: Synergizing reasoning and acting in language models
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao · 2023
Earlier work this paper cites.
Taskweaver: A code-first agent framework
Bo Qiao, Liqun Li, Xu Zhang, Shilin He, Yu Kang, Chaoyun Zhang, Fangkai Yang, Hang Dong, Jue Zhang, Lu Wang, et al · 2023
Earlier work this paper cites.
Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v
Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao · 2023
Earlier work this paper cites.
Real-time flying object detection with yolov8
Dillon Reis, Jordan Kupec, Jacqueline Hong, and Ahmad Daoudi · 2023
Earlier work this paper cites.
Retrieval-augmented generation for large language models: A survey
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Haofen Wang, and Haofen Wang · 2023
Earlier work this paper cites.
Llm as os, agents as apps: Envisioning aios, agents and the aios-agent ecosystem
Yingqiang Ge, Yujie Ren, Wenyue Hua, Shuyuan Xu, Juntao Tan, and Yongfeng Zhang · 2023
Earlier work this paper cites.
Large language model-brained gui agents: A survey
Chaoyun Zhang, Shilin He, Jiaxu Qian, Bowen Li, Liqun Li, Si Qin, Yu Kang, Minghua Ma, Guyue Liu, Qingwei Lin, et al · 2024
Earlier work this paper cites.
Mm-llms: Recent advances in multimodal large language models
Duzhen Zhang, Yahan Yu, Jiahua Dong, Chenxing Li, Dan Su, Chenhui Chu, and Dong Yu · 2024
Cited alongside, same era.
Ufo: A ui-focused agent for windows os interaction
Chaoyun Zhang, Liqun Li, Shilin He, Xu Zhang, Bo Qiao, Si Qin, Minghua Ma, Yu Kang, Qingwei Lin, Saravan Rajmohan, et al · 2024
Cited alongside, same era.
Introducing computer use, a new claude 3.5 sonnet, and claude 3.5 haiku, 2024
Anthropic · 2024
Cited alongside, same era.
Large action models: From inception to implementation
Lu Wang, Fangkai Yang, Chaoyun Zhang, Junting Lu, Jiaxu Qian, Shilin He, Pu Zhao, Bo Qiao, Ray Huang, Si Qin, et al · 2024
Cited alongside, same era.
Screenagent: a vision language model-driven computer control agent
Runliang Niu, Jindong Li, Shiqi Wang, Yali Fu, Xiyu Hu, Xueyuan Leng, He Kong, Yi Chang, and Qi Wang · 2024
Mllm-as-a-judge: Assessing multimodal llm-as-a-judge with vision-language benchmark
Dongping Chen, Ruoxi Chen, Shilin Zhang, Yaochen Wang, Yinuo Liu, Huichi Zhou, Qihui Zhang, Yao Wan, Pan Zhou, and Lichao Sun · 2024
Later among the works it cites.
Windows agent arena: Evaluating multi-modal os agents at scale
Rogerio Bonatti, Dan Zhao, Dillon Dupont, Sara Abdali, Yinheng Li, Yadong Lu, Justin Wagle, Kazuhito Koishida, Arthur Bucker, Lawrence Keunho Jang, et al · 2024
Later among the works it cites.
Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Jing Hua Toh, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al · 2024
Later among the works it cites.
Cogagent: A visual language model for gui agents
Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al · 2024
Later among the works it cites.
Cogvlm: Visual expert for pretrained language models
Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Song XiXuan, et al · 2024
Later among the works it cites.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Cited alongside, same era.
Omniparser for pure vision based gui agent
Yadong Lu, Jianwei Yang, Yelong Shen, and Ahmed Awadallah · 2024
Cited alongside, same era.
Llm multi-agent systems: Challenges and open problems
Shanshan Han, Qifan Zhang, Yuhang Yao, Weizhao Jin, Zhaozhuo Xu, and Chaoyang He · 2024
Cited alongside, same era.
A survey on large language model based autonomous agents
Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al · 2024
Cited alongside, same era.
A survey on the memory mechanism of large language model based agents
Zeyu Zhang, Xiaohe Bo, Chen Ma, Rui Li, Xu Chen, Quanyu Dai, Jieming Zhu, Zhenhua Dong, and Ji-Rong Wen · 2024
Cited alongside, same era.
Everything of thoughts: Defying the law of penrose triangle for thought generation
Ruomeng Ding, Chaoyun Zhang, Lu Wang, Yong Xu, Minghua Ma, Wei Zhang, Si Qin, Saravan Rajmohan, Qingwei Lin, and Dongmei Zhang · 2024
Cited alongside, same era.
Seeclick: Harnessing gui grounding for advanced visual gui agents
Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Li YanTao, Jianbing Zhang, and Zhiyong Wu · 2024
Cited alongside, same era.
Navigating the digital world as humans do: Universal visual grounding for gui agents
Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su · 2024
Cited alongside, same era.
Aios: Llm agent operating system
Kai Mei, Xi Zhu, Wujiang Xu, Wenyue Hua, Mingyu Jin, Zelong Li, Shuyuan Xu, Ruosong Ye, Yingqiang Ge, and Yongfeng Zhang · 2024
Later among the works it cites.
Autoos: make your os more powerful by exploiting large language models
Huilai Chen, Yuanbo Wen, Limin Cheng, Shouxu Kuang, Yumeng Liu, Weijia Li, Ling Li, Rui Zhang, Xinkai Song, Wei Li, et al · 2024
Later among the works it cites.
Uipath: Automation platform, 2025
UiPath · 2025
Closest in time.
Automation anywhere: Automation 360 platform, 2025
Automation Anywhere · 2025
Closest in time.
Microsoft power automate, 2025
Microsoft · 2025
Closest in time.
Computer-using agent: Introducing a universal interface for ai to interact with the digital world
OpenAI · 2025
Closest in time.
Ui-tars: Pioneering automated gui interaction with native agents
Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et al · 2025
Closest in time.
Vem: Environment-free exploration for training gui agent with value environment model
Jiani Zheng, Lu Wang, Fangkai Yang, Chaoyun Zhang, Lingrui Mei, Wenjie Yin, Qingwei Lin, Dongmei Zhang, Saravan Rajmohan, and Qi Zhang · 2025
Closest in time.
Api agents vs. gui agents: Divergence and convergence
Chaoyun Zhang, Shilin He, Liqun Li, Si Qin, Yu Kang, Qingwei Lin, and Dongmei Zhang · 2025
Closest in time.
Benchmark self-evolving: A multi-agent framework for dynamic llm evaluation
Siyuan Wang, Zhuohan Long, Zhihao Fan, Xuan-Jing Huang, and Zhongyu Wei · 2025
Closest in time.
Ui-r1: Enhancing action prediction of gui agents by reinforcement learning, 2025
Zhengxi Lu, Yuxiang Chai, Yaxuan Guo, Xi Yin, Liang Liu, Hao Wang, Guanjing Xiong, and Hongsheng Li · 2025
Closest in time.
At-spi - assistive technology service provider interface
Linux From Scratch · 2025
Closest in time.
Accessibility api
Apple Inc · 2025
Closest in time.
Cerebrum (aios sdk): A platform for agent development, deployment, distribution, and discovery, 2025
Balaji Rama, Kai Mei, and Yongfeng Zhang · 2025
Closest in time.