Fetching the paper…
Reading the bibliography…
The emergent large language/multimodal models facilitate the evolution of mobile agents, especially in mobile UI task automation.
XML Path Language (XPath) 3.1
2017 · 2017
Earlier work this paper cites.
Rico: A mobile app dataset for building data-driven design applications. In Proceedings of the 30th annual ACM symposium on user interface software and technology . 845–854
Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. 2017 · 2017
Earlier work this paper cites.
Single activity: Why, when, and how (Android Dev Summit ’18)
2018 · 2018
Earlier work this paper cites.
Mapping natural language instructions to mobile UI action sequences
Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. 2020 · 2020
Earlier work this paper cites.
Exploring the limits of transfer learning with a unified text-to-text transformer
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020 · 2020
Earlier work this paper cites.
UIED: a hybrid tool for GUI element detection. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering . 1655–1659
Mulong Xie, Sidong Feng, Zhenchang Xing, Jieshan Chen, and Chunyang Chen. 2020 · 2020
Earlier work this paper cites.
Vins: Visual search for mobile user interface design. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems . 1–14
Sara Bunian, Kai Li, Chaima Jemmali, Casper Harteveld, Yun Fu, and Magy Seif Seif El-Nasr. 2021 · 2021
Earlier work this paper cites.
Androidenv: A reinforcement learning platform for android
Daniel Toyama, Philippe Hamel, Anita Gergely, Gheorghe Comanici, Amelia Glaese, Zafarali Ahmed, Tyler Jackson, Shibl Mourad, and Doina Precup. 2021 · 2021
Earlier work this paper cites.
A dataset for interactive vision-language navigation with unknown command feasibility. In European Conference on Computer Vision . Springer, 312–328
Andrea Burns, Deniz Arsan, Sanjna Agrawal, Ranjitha Kumar, Kate Saenko, and Bryan A Plummer. 2022 · 2022
Earlier work this paper cites.
Understanding screen relationships from screenshots of smartphone applications. In 27th International Conference on Intelligent User Interfaces . 447–458
Shirin Feiz, Jason Wu, Xiaoyi Zhang, Amanda Swearngin, Titus Barik, and Jeffrey Nichols. 2022 · 2022
Earlier work this paper cites.
META-GUI: Towards Multi-modal Conversational Agents on Mobile GUI
Liangtai Sun, Xingyu Chen, Lu Chen, Tianle Dai, Zichen Zhu, and Kai Yu. 2022 · 2022
Earlier work this paper cites.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al · 2023
Earlier work this paper cites.
Can Large Language Models Be an Alternative to Human Evaluations?
Cheng-Han Chiang and Hung-yi Lee. 2023 · 2023
Cited alongside, same era.
Run apps on the Android Emulator | Android Developers
Google. 2023 · 2023
Cited alongside, same era.
CogAgent: A Visual Language Model for GUI Agents
Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al · 2023
Cited alongside, same era.
Sunjae Lee, Junyoung Choi, Jungjae Lee, Hojun Choi, Steven Y Ko, Sangeun Oh, and Insik Shin. 2023 · 2023
Cited alongside, same era.
GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigation
An Yan, Zhengyuan Yang, Wanrong Zhu, Kevin Lin, Linjie Li, Jianfeng Wang, Jianwei Yang, Yiwu Zhong, Julian McAuley, Jianfeng Gao, et al · 2023
Later among the works it cites.
AppAgent: Multimodal Agents as Smartphone Users
Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. 2023 · 2023
Later among the works it cites.
You only look at screens: Multimodal chain-of-action agents
Zhuosheng Zhan and Aston Zhang. 2023 · 2023
Later among the works it cites.
Mobile-env: A universal platform for training and evaluation of mobile interaction
Danyang Zhang, Lu Chen, and Kai Yu. 2023 · 2023
Later among the works it cites.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023 · 2023
Cited alongside, same era.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023 · 2023
Cited alongside, same era.
GPT-4V(ision) system card
OpenAI. 2023 · 2023
Cited alongside, same era.
AutoTask: Executing Arbitrary Voice Commands by Exploring and Learning from Mobile GUI
Lihang Pan, Bowen Wang, Chun Yu, Yuxuan Chen, Xiangyu Zhang, and Yuanchun Shi. 2023 · 2023
Cited alongside, same era.
Android in the wild: A large-scale dataset for android device control
Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. 2023 · 2023
Cited alongside, same era.
AXNav: Replaying Accessibility Tests from Natural Language
Maryam Taeb, Amanda Swearngin, Eldon School, Ruijia Cheng, Yue Jiang, and Jeffrey Nichols. 2023 · 2023
Cited alongside, same era.
Enabling conversational interaction with mobile ui using large language models. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems . 1–17
Bryan Wang, Gang Li, and Yang Li. 2023 · 2023
Cited alongside, same era.
WebUI: A Dataset for Enhancing Visual UI Understanding with Web Semantics. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems . 1–14
Jason Wu, Siyan Wang, Siman Shen, Yi-Hao Peng, Jeffrey Nichols, and Jeffrey P Bigham. 2023b · 2023
Cited alongside, same era.
Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, et al · 2023
Later among the works it cites.
Activity | Android Developers
2024 · 2024
Closest in time.
Android UIAutomator2
2024 · 2024
Closest in time.
Siri - Apple
Apple. 2024 · 2024
Closest in time.
Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Security
Yuanchun Li, Hao Wen, Weijun Wang, Xiangyu Li, Yizhen Yuan, Guohong Liu, Jiacheng Liu, Wenxing Xu, Xiang Wang, Yi Sun, et al · 2024
Closest in time.
Comprehensive Cognitive LLM Agent for Smartphone GUI Automation
Xinbei Ma, Zhuosheng Zhang, and Hai Zhao. 2024 · 2024
Closest in time.
Autodroid: Llm-powered task automation in android
Hao Wen, Yuanchun Li, Guohong Liu, Shanhui Zhao, Tao Yu, Toby Jia-Jun Li, Shiqi Jiang, Yunhao Liu, Yaqin Zhang, and Yunxin Liu. 2024 · 2024
Closest in time.
Understanding the Weakness of Large Language Model Agents within a Complex Android Environment
Mingzhe Xing, Rongkai Zhang, Hui Xue, Qi Chen, Fan Yang, and Zhen Xiao. 2024 · 2024
Closest in time.