Fetching the paper…
Reading the bibliography…
Phone automation agents aim to autonomously perform a given natural-language user request, such as scheduling appointments or booking a hotel.
A survey on dialogue systems: Recent advances and new frontiers
Hongshen Chen, Xiaorui Liu, Dawei Yin, and Jiliang Tang. 2017 · 2017
Earlier work this paper cites.
Mapping natural language instructions to mobile UI action sequences
Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. 2020 · 2020
Earlier work this paper cites.
Conversations with GUIs. In Proceedings of the 2021 ACM Designing Interactive Systems Conference . 1447–1457
Kashyap Todi, Luis A Leiva, Daniel Buschek, Pin Tian, and Antti Oulasvirta. 2021 · 2021
Earlier work this paper cites.
Androidenv: A reinforcement learning platform for android
Daniel Toyama, Philippe Hamel, Anita Gergely, Gheorghe Comanici, Amelia Glaese, Zafarali Ahmed, Tyler Jackson, Shibl Mourad, and Doina Precup. 2021 · 2021
Earlier work this paper cites.
Screen2words: Automatic mobile UI summarization with multimodal learning. In The 34th Annual ACM Symposium on User Interface Software and Technology . 498–510
Bryan Wang, Gang Li, Xin Zhou, Zhourong Chen, Tovi Grossman, and Yang Li. 2021 · 2021
Earlier work this paper cites.
Screen recognition: Creating accessibility metadata for mobile applications from pixels. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems . 1–15
Xiaoyi Zhang, Lilian De Greef, Amanda Swearngin, Samuel White, Kyle Murray, Lisa Yu, Qi Shan, Jeffrey Nichols, Jason Wu, Chris Fleizach, et al · 2021
Earlier work this paper cites.
A dataset for interactive vision-language navigation with unknown command feasibility. In European Conference on Computer Vision . Springer, 312–328
Andrea Burns, Deniz Arsan, Sanjna Agrawal, Ranjitha Kumar, Kate Saenko, and Bryan A Plummer. 2022 · 2022
Earlier work this paper cites.
mobile-env: An open platform for reinforcement learning in wireless mobile networks. In NOMS 2022-2022 IEEE/IFIP Network Operations and Management Symposium . IEEE, 1–3
Stefan Schneider, Stefan Werner, Ramin Khalili, Artur Hecker, and Holger Karl. 2022 · 2022
Earlier work this paper cites.
META-GUI: towards multi-modal conversational agents on mobile GUI
Liangtai Sun, Xingyu Chen, Lu Chen, Tianle Dai, Zichen Zhu, and Kai Yu. 2022 · 2022
Earlier work this paper cites.
UGIF: UI Grounded Instruction Following
Sagar Gubbi Venkatesh, Partha Talukdar, and Srini Narayanan. 2022 · 2022
Earlier work this paper cites.
Chain-of-thought prompting elicits reasoning in large language models
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al · 2022
Cited alongside, same era.
Mind2Web: Towards a Generalist Agent for the Web
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. 2023 · 2023
Cited alongside, same era.
Cogagent: A visual language model for gui agents
Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al · 2023
Cited alongside, same era.
UINav: A maker of UI automation agents
Wei Li, Fu-Lin Hsu, Will Bishop, Folawiyo Campbell-Ajala, Oriana Riva, and Max Lin. 2023 · 2023
Cited alongside, same era.
Android in the wild: A large-scale dataset for android device control
Appagent: Multimodal agents as smartphone users
Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. 2023 · 2023
Later among the works it cites.
You only look at screens: Multimodal chain-of-action agents
Zhuosheng Zhan and Aston Zhang. 2023 · 2023
Later among the works it cites.
Latent State Estimation Helps UI Agents to Reason
William E Bishop, Alice Li, Christopher Rawles, and Oriana Riva. 2024 · 2024
Later among the works it cites.
Aligning Language Models to Explicitly Handle Ambiguity
Hyuhng Joon Kim, Youna Kim, Cheonbok Park, Junyeob Kim, Choonghyun Park, Kang Min Yoo, Sang-goo Lee, and Taeuk Kim. 2024 · 2024
Later among the works it cites.
Personal llm agents: Insights and survey about the capability, efficiency and security
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. 2023 · 2023
Cited alongside, same era.
Robots that ask for help: Uncertainty alignment for large language model planners
Allen Z Ren, Anushri Dixit, Alexandra Bodrova, Sumeet Singh, Stephen Tu, Noah Brown, Peng Xu, Leila Takayama, Fei Xia, Jake Varley, et al · 2023
Cited alongside, same era.
Enabling conversational interaction with mobile ui using large language models. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems . 1–17
Bryan Wang, Gang Li, and Yang Li. 2023 · 2023
Cited alongside, same era.
Empowering llm to use smartphone for intelligent task automation
Hao Wen, Yuanchun Li, Guohong Liu, Shanhui Zhao, Tao Yu, Toby Jia-Jun Li, Shiqi Jiang, Yunhao Liu, Yaqin Zhang, and Yunxin Liu. 2023 · 2023
Cited alongside, same era.
Gpt-4v in wonderland: Large multimodal models for zero-shot smartphone gui navigation
An Yan, Zhengyuan Yang, Wanrong Zhu, Kevin Lin, Linjie Li, Jianfeng Wang, Jianwei Yang, Yiwu Zhong, Julian McAuley, Jianfeng Gao, et al · 2023
Cited alongside, same era.
Yuanchun Li, Hao Wen, Weijun Wang, Xiangyu Li, Yizhen Yuan, Guohong Liu, Jiacheng Liu, Wenxing Xu, Xiang Wang, Yi Sun, et al · 2024
Later among the works it cites.
Weblinx: Real-world website navigation with multi-turn dialogue
Xing Han Lù, Zdeněk Kasner, and Siva Reddy. 2024 · 2024
Later among the works it cites.
Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al · 2024
Later among the works it cites.
On the effects of data scale on ui control agents
Wei Li, William E Bishop, Alice Li, Christopher Rawles, Folawiyo Campbell-Ajala, Divya Tyamagundlu, and Oriana Riva. 2025 · 2025
Closest in time.
Recent advances and challenges in task-oriented dialog systems
Zheng Zhang, Ryuichi Takanobu, Qi Zhu, MinLie Huang, and XiaoYan Zhu. 2020 · 2027
Closest in time.