Fetching the paper…
Reading the bibliography…
The dream to create AI assistants as capable and versatile as the fictional J.A.R.V.I.S from Iron Man has long captivated imaginations.
World of bits: An open-domain platform for web-based agents
Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang · 2017
Earlier work this paper cites.
Bert: Pre-training of deep bidirectional transformers for language understanding
Jacob Devlin · 2018
Earlier work this paper cites.
Reinforcement learning: An introduction
Richard S Sutton · 2018
Earlier work this paper cites.
Reinforcement learning on web interfaces using workflow-guided exploration
Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, and Percy Liang · 2018
Earlier work this paper cites.
Izzeddin Gur, Ulrich Rueckert, Aleksandra Faust, and Dilek Hakkani-Tur · 2018
Earlier work this paper cites.
Mapping natural language commands to web elements
Panupong Pasupat, Tian-Shun Jiang, Evan Zheran Liu, Kelvin Guu, and Percy Liang · 2018
Earlier work this paper cites.
Survey on virtual assistant: Google assistant, siri, cortana, alexa
Amrita S Tulshan and Sudhir Namdeorao Dhage · 2019
Earlier work this paper cites.
Dom-q-net: Grounded rl on structured language
Sheng Jia, Jamie Kiros, and Jimmy Ba · 2019
Earlier work this paper cites.
Language models are few-shot learners
Tom B Brown · 2020
Earlier work this paper cites.
An image is worth 16x16 words: Transformers for image recognition at scale
Alexey Dosovitskiy · 2020
Earlier work this paper cites.
Mapping natural language instructions to mobile ui action sequences
Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge · 2020
Earlier work this paper cites.
Lawformer: A pre-trained language model for chinese legal long documents
Chaojun Xiao, Xueyu Hu, Zhiyuan Liu, Cunchao Tu, and Maosong Sun · 2021
Earlier work this paper cites.
Appbuddy: Learning to accomplish tasks in mobile apps via reinforcement learning
Maayan Shvo, Zhiming Hu, Rodrigo Toro Icarte, Iqbal Mohomed, Allan D Jepson, and Sheila A McIlraith · 2021
Earlier work this paper cites.
Ugif: Ui grounded instruction following
Sagar Gubbi Venkatesh, Partha Talukdar, and Srini Narayanan · 2022
Earlier work this paper cites.
Webshop: Towards scalable real-world web interaction with grounded language agents
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan · 2022
Earlier work this paper cites.
Meta-gui: Towards multi-modal conversational agents on mobile gui
Liangtai Sun, Xingyu Chen, Lu Chen, Tianle Dai, Zichen Zhu, and Kai Yu · 2022
Earlier work this paper cites.
LangChain, October 2022
Harrison Chase · 2022
Earlier work this paper cites.
A data-driven approach for learning to control computers
Peter C Humphreys, David Raposo, Tobias Pohlen, Gregory Thornton, Rachita Chhaparia, Alistair Muldal, Josh Abramson, Petko Georgiev, Adam Santoro, and Timothy Lillicrap · 2022
Earlier work this paper cites.
Towards better semantic understanding of mobile interfaces
Srinivas Sunkara, Maria Wang, Lijuan Liu, Gilles Baechler, Yu-Chung Hsiao, Abhanshu Sharma, James Stout, et al · 2022
Earlier work this paper cites.
A dataset for interactive vision-language navigation with unknown command feasibility
Andrea Burns, Deniz Arsan, Sanjna Agrawal, Ranjitha Kumar, Kate Saenko, and Bryan A Plummer · 2022
Earlier work this paper cites.
Yong Dai, Duyu Tang, Liangxin Liu, Minghuan Tan, Cong Zhou, Jingquan Wang, Zhangyin Feng, Fan Zhang, Xueyu Hu, and Shuming Shi · 2022
Earlier work this paper cites.
Reasoning with language model prompting: A survey
Shuofei Qiao, Yixin Ou, Ningyu Zhang, Xiang Chen, Yunzhi Yao, Shumin Deng, Chuanqi Tan, Fei Huang, and Huajun Chen · 2022
Earlier work this paper cites.
A real-world webagent with planning, long context understanding, and program synthesis
Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust · 2023
Earlier work this paper cites.
Gpt-4v in wonderland: Large multimodal models for zero-shot smartphone gui navigation
An Yan, Zhengyuan Yang, Wanrong Zhu, Kevin Lin, Linjie Li, Jianfeng Wang, Jianwei Yang, Yiwu Zhong, Julian McAuley, Jianfeng Gao, et al · 2023
Earlier work this paper cites.
Laser: Llm agent with state-space exploration for web navigation
Kaixin Ma, Hongming Zhang, Hongwei Wang, Xiaoman Pan, Wenhao Yu, and Dong Yu · 2023
Earlier work this paper cites.
Assistgui: Task-oriented desktop graphical user interface automation
Difei Gao, Lei Ji, Zechen Bai, Mingyu Ouyang, Peiran Li, Dongxing Mao, Qinchen Wu, Weichen Zhang, Peiyi Wang, Xiangwu Guo, et al · 2023
Earlier work this paper cites.
You only look at screens: Multimodal chain-of-action agents
Zhuosheng Zhang and Aston Zhang · 2023
Earlier work this paper cites.
Towards reasoning in large language models: A survey, 2023
Jie Huang and Kevin Chen-Chuan Chang · 2023
Earlier work this paper cites.
React: Synergizing reasoning and acting in language models, 2023
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao · 2023
Earlier work this paper cites.
Code llama: Open foundation models for code
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al · 2023
Earlier work this paper cites.
Bloomberggpt: A large language model for finance
Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann · 2023
Earlier work this paper cites.
Towards expert-level medical question answering with large language models
Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Le Hou, Kevin Clark, Stephen Pfohl, Heather Cole-Lewis, Darlene Neal, et al · 2023
Earlier work this paper cites.
Iluvui: Instruction-tuned language-vision modeling of uis from machine conversations
Yue Jiang, Eldon Schoop, Amanda Swearngin, and Jeffrey Nichols · 2023
Earlier work this paper cites.
Multimodal web navigation with instruction-finetuned foundation models
Hiroki Furuta, Kuang-Huei Lee, Ofir Nachum, Yutaka Matsuo, Aleksandra Faust, Shixiang Shane Gu, and Izzeddin Gur · 2023
Earlier work this paper cites.
From pixels to ui actions: Learning to follow instructions via graphical user interfaces
Peter Shaw, Mandar Joshi, James Cohan, Jonathan Berant, Panupong Pasupat, Hexiang Hu, Urvashi Khandelwal, Kenton Lee, and Kristina N Toutanova · 2023
Earlier work this paper cites.
Webwise: Web interface control and sequential exploration with large language models
Heyi Tao, Sethuraman TV, Michal Shlapentokh-Rothman, and Derek Hoiem · 2023
Earlier work this paper cites.
A zero-shot language agent for computer control with structured reflection
Tao Li, Gang Li, Zhiwei Deng, Bryan Wang, and Yang Li · 2023
Earlier work this paper cites.
Enabling conversational interaction with mobile ui using large language models
Bryan Wang, Gang Li, and Yang Li · 2023
Earlier work this paper cites.
Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v, 2023
Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao · 2023
Earlier work this paper cites.
Chain-of-thought prompting elicits reasoning in large language models, 2023
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou · 2023
Earlier work this paper cites.
Reflexion: Language agents with verbal reinforcement learning, 2023
Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao · 2023
Earlier work this paper cites.
Agentbench: Evaluating llms as agents
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al · 2023
Earlier work this paper cites.
Pptc benchmark: Evaluating large language models for powerpoint task completion
Yiduo Guo, Zekai Zhang, Yaobo Liang, Dongyan Zhao, and Nan Duan · 2023
Earlier work this paper cites.
Survey of vulnerabilities in large language models revealed by adversarial attacks, 2023
Erfan Shayegani, Md Abdullah Al Mamun, Yu Fu, Pedram Zaree, Yue Dong, and Nael Abu-Ghazaleh · 2023
Earlier work this paper cites.
Rodrigo Pedro, Daniel Castro, Paulo Carreira, and Nuno Santos · 2023
Cited alongside, same era.
Xizhou Zhu, Yuntao Chen, Hao Tian, Chenxin Tao, Weijie Su, Chenyu Yang, Gao Huang, Bin Li, Lewei Lu, Xiaogang Wang, et al · 2023
Cited alongside, same era.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou · 2023
Cited alongside, same era.
A survey of large language models
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al · 2023
Cited alongside, same era.
Exposing limitations of language model agents in sequential-task compositions on the web
Hiroki Furuta, Yutaka Matsuo, Aleksandra Faust, and Izzeddin Gur · 2024
Later among the works it cites.
AGILE: A novel reinforcement learning framework of LLM agents
FengPeiyuan, Yichen He, Guanhua Huang, Yuan Lin, Hanchong Zhang, Yuchen Zhang, and Hang Li · 2024
Later among the works it cites.
Digirl: Training in-the-wild device-control agents with autonomous reinforcement learning
Hao Bai, Yifei Zhou, Mert Cemri, Jiayi Pan, Alane Suhr, Sergey Levine, and Aviral Kumar · 2024
Later among the works it cites.
Large language models empowered personalized web agents
Hongru Cai, Yongqi Li, Wenjie Wang, Fengbin Zhu, Xiaoyu Shen, Wenjie Li, and Tat-Seng Chua · 2024
Later among the works it cites.
Agent s: An open agentic framework that uses computers like a human
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Large language models for software engineering: A systematic literature review
Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang · 2023
Cited alongside, same era.
Siri - apple, 2024
Apple Inc · 2024
Cited alongside, same era.
Cortana research - microsoft research, 2024
Microsoft Research · 2024
Cited alongside, same era.
Google assistant, 2024
Google · 2024
Cited alongside, same era.
Alexa - amazon, 2024
Amazon · 2024
Cited alongside, same era.
Gemini - google
Google · 2024
Cited alongside, same era.
Home - openai
OpenAI · 2024
Cited alongside, same era.
Anthropic
Anthropic · 2024
Cited alongside, same era.
Saaket Agashe, Jiuzhou Han, Shuyu Gan, Jiachen Yang, Ang Li, and Xin Eric Wang · 2024
Later among the works it cites.
From commands to prompts: Llm-based semantic file system for aios
Zeru Shi, Kai Mei, Mingyu Jin, Yongye Su, Chaoji Zuo, Wenyue Hua, Wujiang Xu, Yujie Ren, Zirui Liu, Mengnan Du, et al · 2024
Later among the works it cites.
Naviqate: Functionality-guided web application navigation, 2024
Mobina Shahbandeh, Parsa Alian, Noor Nashid, and Ali Mesbah · 2024
Later among the works it cites.
Towards llmci-multimodal ai for llm-vision ui operation
Husam Barham and Mohammed Fasha · 2024
Later among the works it cites.
Agent-e: From autonomous web navigation to foundational design principles in agentic systems
Tamer Abuelsaad, Deepak Akkil, Prasenjit Dey, Ashish Jagmohan, Aditya Vempaty, and Ravi Kokku · 2024
Later among the works it cites.
Cradle: Empowering foundation agents towards general computer control
Weihao Tan, Wentao Zhang, Xinrun Xu, Haochong Xia, Gang Ding, Boyu Li, Bohan Zhou, Junpeng Yue, Jiechuan Jiang, Yewen Li, et al · 2024
Later among the works it cites.
Mobileagent: enhancing mobile control via human-machine interaction and sop integration
Tinghe Ding · 2024
Later among the works it cites.
Coco-agent: A comprehensive cognitive mllm agent for smartphone gui automation
Xinbei Ma, Zhuosheng Zhang, and Hai Zhao · 2024
Later among the works it cites.
Infogent: An agent-based framework for web information aggregation
Revanth Gangi Reddy, Sagnik Mukherjee, Jeonghwan Kim, Zhenhailong Wang, Dilek Hakkani-Tur, and Heng Ji · 2024
Later among the works it cites.
Moba: A two-level agent system for efficient mobile task automation
Zichen Zhu, Hao Tang, Yansi Li, Kunyao Lan, Yixuan Jiang, Hao Zhou, Yixiao Wang, Situo Zhang, Liangtai Sun, Lu Chen, et al · 2024
Later among the works it cites.
Screenagent: A vision language model-driven computer control agent
Runliang Niu, Jindong Li, Shiqi Wang, Yali Fu, Xiyu Hu, Xueyuan Leng, He Kong, Yi Chang, and Qi Wang · 2024
Later among the works it cites.
Caap: Context-aware action planning prompting to solve computer tasks with front-end ui only
Junhee Cho, Jihoon Kim, Daseul Bae, Jinho Choo, Youngjune Gwon, and Yeong-Dae Kwon · 2024
Later among the works it cites.
Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al · 2024
Later among the works it cites.
Webcanvas: Benchmarking web agents in online environments
Yichen Pan, Dehan Kong, Sida Zhou, Cheng Cui, Yifei Leng, Bing Jiang, Hangyu Liu, Yanyi Shang, Shuyan Zhou, Tongshuang Wu, et al · 2024
Later among the works it cites.
Kaining Ying, Fanqing Meng, Jin Wang, Zhiqian Li, Han Lin, Yue Yang, Hao Zhang, Wenbo Zhang, Yuqi Lin, Shuo Liu, et al · 2024
Later among the works it cites.
Shopping mmlu: A massive multi-task online shopping benchmark for large language models
Yilun Jin, Zheng Li, Chenwei Zhang, Tianyu Cao, Yifan Gao, Pratik Jayarao, Mao Li, Xin Liu, Ritesh Sarkhel, Xianfeng Tang, et al · 2024
Later among the works it cites.
Foundational autoraters: Taming large language models for better automatic evaluation, 2024
Tu Vu, Kalpesh Krishna, Salaheddin Alzubi, Chris Tar, Manaal Faruqui, and Yun-Hsuan Sung · 2024
Later among the works it cites.
A survey on llm-as-a-judge, 2024
Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Yuanzhuo Wang, and Jian Guo · 2024
Later among the works it cites.
Visual grounding for desktop graphical user interfaces
Tassnim Dardouri, Laura Minkova, Jessica López Espejel, Walid Dahhane, and El Hassane Ettifouri · 2024
Later among the works it cites.
Large language models as software components: A taxonomy for llm-integrated applications
Irene Weber · 2024
Later among the works it cites.
Spa-bench: A comprehensive benchmark for smartphone agent evaluation
Jingxuan Chen, Derek Yuen, Bin Xie, Yuhao Yang, Gongwei Chen, Zhihao Wu, Li Yixing, Xurui Zhou, Weiwen Liu, Shuai Wang, et al · 2024
Later among the works it cites.
Human player outwits freysa ai agent in $47,000 crypto challenge, 2024
Danny Park · 2024
Later among the works it cites.
A survey on large language model (llm) security and privacy: The good, the bad, and the ugly
Yifan Yao, Jinhao Duan, Kaidi Xu, Yuanfang Cai, Zhibo Sun, and Yue Zhang · 2024
Later among the works it cites.
Risk taxonomy, mitigation, and assessment benchmarks of large language model systems, 2024
Tianyu Cui, Yanling Wang, Chuanpu Fu, Yong Xiao, Sijia Li, Xinhao Deng, Yunpeng Liu, Qinglin Zhang, Ziyi Qiu, Peiyang Li, Zhixing Tan, Junwu Xiong, Xinyu Kong, Zujie Wen, Ke Xu, and Qi Li · 2024
Later among the works it cites.
Boosting llm agents with recursive contemplation for effective deception handling
Shenzhi Wang, Chang Liu, Zilong Zheng, Siyuan Qi, Shuo Chen, Qisen Yang, Andrew Zhao, Chaofei Wang, Shiji Song, and Gao Huang · 2024
Later among the works it cites.
Privacy issues in large language models: A survey, 2024
Seth Neel and Peter Chang · 2024
Later among the works it cites.
Eia: Environmental injection attack on generalist web agents for privacy leakage, 2024
Zeyi Liao, Lingbo Mo, Chejian Xu, Mintong Kang, Jiawei Zhang, Chaowei Xiao, Yuan Tian, Bo Li, and Huan Sun · 2024
Later among the works it cites.
Refusal-trained llms are easily jailbroken as browser agents, 2024
Priyanshu Kumar, Elaine Lau, Saranya Vijayakumar, Tu Trinh, Scale Red Team, Elaine Chang, Vaughn Robinson, Sean Hendryx, Shuyan Zhou, Matt Fredrikson, Summer Yue, and Zifan Wang · 2024
Later among the works it cites.
Identifying the risks of lm agents with an lm-emulated sandbox, 2024
Yangjun Ruan, Honghua Dong, Andrew Wang, Silviu Pitis, Yongchao Zhou, Jimmy Ba, Yann Dubois, Chris J. Maddison, and Tatsunori Hashimoto · 2024
Later among the works it cites.
Trustagent: Towards safe and trustworthy llm-based agents, 2024
Wenyue Hua, Xianjun Yang, Mingyu Jin, Zelong Li, Wei Cheng, Ruixiang Tang, and Yongfeng Zhang · 2024
Later among the works it cites.
Haishuo Fang, Xiaodan Zhu, and Iryna Gurevych · 2024
Later among the works it cites.
Guardagent: Safeguard llm agents by a guard agent via knowledge-enabled reasoning, 2024
Zhen Xiang, Linzhi Zheng, Yanjie Li, Junyuan Hong, Qinbin Li, Han Xie, Jiawei Zhang, Zidi Xiong, Chulin Xie, Carl Yang, Dawn Song, and Bo Li · 2024
Later among the works it cites.
Md Shamsujjoha, Qinghua Lu, Dehai Zhao, and Liming Zhu · 2024
Later among the works it cites.
St-webagentbench: A benchmark for evaluating safety and trustworthiness in web agents, 2024
Ido Levy, Ben Wiesel, Sami Marreed, Alon Oved, Avi Yaeli, and Segev Shlomov · 2024
Later among the works it cites.
Symbolic learning enables self-evolving agents, 2024
Wangchunshu Zhou, Yixin Ou, Shengwei Ding, Long Li, Jialong Wu, Tiannan Wang, Jiamin Chen, Shuai Wang, Xiaohua Xu, Ningyu Zhang, Huajun Chen, and Yuchen Eleanor Jiang · 2024
Later among the works it cites.
Alan Wake, Albert Wang, Bei Chen, CX Lv, Chao Li, Chengen Huang, Chenglin Cai, Chujie Zheng, Daniel Cooper, Ethan Dai, et al · 2024
Later among the works it cites.
A survey on multimodal large language models
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen · 2024
Later among the works it cites.
On llms-driven synthetic data generation, curation, and evaluation: A survey
Lin Long, Rui Wang, Ruixuan Xiao, Junbo Zhao, Xiao Ding, Gang Chen, and Haobo Wang · 2024
Later among the works it cites.
Ferret-ui: Grounded mobile ui understanding with multimodal llms
Keen You, Haotian Zhang, Eldon Schoop, Floris Weers, Amanda Swearngin, Jeffrey Nichols, Yinfei Yang, and Zhe Gan · 2025
Closest in time.