MobileSteward: Integrating Multiple App-Oriented Agents with Self-Evolution to Automate Cross-App Instructions · Around

Fetching the paper…

MobileSteward: Integrating Multiple App-Oriented Agents with Self-Evolution to Automate Cross-App Instructions · Around

Built on

K. Lieberherr, I. Holland, and A. Riel, “Object-oriented programming: An objective sense of style,” ACM Sigplan Notices , vol. 23, no. 11, pp. 323–334, 1988
1988
Earlier work this paper cites.
S. Robertson, H. Zaragoza et al. , “The probabilistic relevance framework: Bm25 and beyond,” Foundations and Trends® in Information Retrieval , vol. 3, no. 4, pp. 333–389, 2009
2009
Earlier work this paper cites.
Y. Li, J. He, X. Zhou, Y. Zhang, and J. Baldridge, “Mapping natural language instructions to mobile ui action sequences,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , 2020, pp. 8198–8210
2020
Earlier work this paper cites.
Y. Li, G. Li, L. He, J. Zheng, H. Li, and Z. Guan, “Widget captioning: Generating natural language description for mobile user interface elements,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2020, pp. 5495–5510
2020
Earlier work this paper cites.
W. Li, “Learning ui navigation through demonstrations composed of macro actions,” arXiv preprint arXiv:2110.08653 , 2021
Original
2021
Earlier work this paper cites.
B. Wang, G. Li, X. Zhou, Z. Chen, T. Grossman, and Y. Li, “Screen2words: Automatic mobile ui summarization with multimodal learning,” in The 34th Annual ACM Symposium on User Interface Software and Technology , 2021, pp. 498–510
2021
Earlier work this paper cites.
T. J.-J. Li, L. Popowski, T. Mitchell, and B. A. Myers, “Screen2vec: Semantic embedding of gui screens and gui components,” in Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems , 2021, pp. 1–15
2021
Earlier work this paper cites.
X. Zhang, L. De Greef, A. Swearngin, S. White, K. Murray, L. Yu, Q. Shan, J. Nichols, J. Wu, C. Fleizach et al. , “Screen recognition: Creating accessibility metadata for mobile applications from pixels,” in Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems , 2021, pp. 1–15
2021
Earlier work this paper cites.
C. Bai, X. Zang, Y. Xu, S. Sunkara, A. Rastogi, J. Chen et al. , “Uibert: Learning generic multimodal representations for ui understanding,” arXiv preprint arXiv:2107.13731 , 2021
Original
2021
Earlier work this paper cites.
A. Burns, D. Arsan, S. Agrawal, R. Kumar, K. Saenko, and B. A. Plummer, “Interactive mobile app navigation with uncertain or under-specified natural language commands,” arXiv preprint arXiv:2202.02312 , 2022
Original
2022
Earlier work this paper cites.
J. Chen, A. Swearngin, J. Wu, T. Barik, J. Nichols, and X. Zhang, “Towards complete icon labeling in mobile applications,” in Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems , 2022, pp. 1–14
2022
Earlier work this paper cites.
A. Burns, D. Arsan, S. Agrawal, R. Kumar, K. Saenko, and B. A. Plummer, “A dataset for interactive vision-language navigation with unknown command feasibility,” in European Conference on Computer Vision . Springer, 2022, pp. 312–328
2022
Earlier work this paper cites.
S. G. Venkatesh, P. Talukdar, and S. Narayanan, “Ugif: Ui grounded instruction following,” arXiv preprint arXiv:2211.07615 , 2022
Original
2022
Earlier work this paper cites.
L. Sun, X. Chen, L. Chen, T. Dai, Z. Zhu, and K. Yu, “Meta-gui: Towards multi-modal conversational agents on mobile gui,” in Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , 2022, pp. 6699–6712
2022
Earlier work this paper cites.
Y. Guan, D. Wang, Z. Chu, S. Wang, F. Ni, R. Song, L. Li, J. Gu, and C. Zhuang, “Intelligent virtual assistants with llm-based process automation,” arXiv preprint arXiv:2312.06677 , 2023
Original
2023
Earlier work this paper cites.
H. Wen, Y. Li, G. Liu, S. Zhao, T. Yu, T. J.-J. Li, S. Jiang, Y. Liu, Y. Zhang, and Y. Liu, “Empowering llm to use smartphone for intelligent task automation,” arXiv preprint arXiv:2308.15272 , 2023
Original
2023
Earlier work this paper cites.
G. Li and Y. Li, “Spotlight: Mobile ui understanding using vision-language models with a focus,” in The Eleventh International Conference on Learning Representations , 2023
2023
Earlier work this paper cites.
A. Yan, Z. Yang, W. Zhu, K. Lin, L. Li, J. Wang, J. Yang, Y. Zhong, J. McAuley, J. Gao et al. , “Gpt-4v in wonderland: Large multimodal models for zero-shot smartphone gui navigation,” arXiv preprint arXiv:2311.07562 , 2023
Original
2023
Earlier work this paper cites.
B. Wang, G. Li, and Y. Li, “Enabling conversational interaction with mobile ui using large language models,” in Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems , 2023, pp. 1–17
2023
Earlier work this paper cites.
Z. Zhan and A. Zhang, “You only look at screens: Multimodal chain-of-action agents,” arXiv preprint arXiv:2309.11436 , 2023
Original
2023
Earlier work this paper cites.
Z. Yang, J. Liu, Y. Han, X. Chen, Z. Huang, B. Fu, and G. Yu, “Appagent: Multimodal agents as smartphone users,” arXiv preprint arXiv:2312.13771 , 2023
Original
2023
Earlier work this paper cites.

Similar

H. Wen, H. Wang, J. Liu, and Y. Li, “Droidbot-gpt: Gpt-powered ui automation for android,” arXiv preprint arXiv:2304.07061 , 2023
Original
2023
Cited alongside, same era.
S. Lee, J. Choi, J. Lee, H. Choi, S. Y. Ko, S. Oh, and I. Shin, “Explore, select, derive, and recall: Augmenting llm with human-like memory for mobile task automation,” arXiv preprint arXiv:2312.03003 , 2023
Original
2023
Cited alongside, same era.
A. Team, “Autogpt,” 2023. [Online]. Available: https://github.com/Significant-Gravitas/AutoGPT
2023
Cited alongside, same era.
C. Qian and X. Cong, “Communicative agents for software development,” arXiv preprint arXiv:2307.07924 , vol. 6, 2023
Original
2023
Cited alongside, same era.
G. Li, H. Hammoud, H. Itani, D. Khizbullin, and B. Ghanem, “Camel: Communicative agents for" mind" exploration of large language model society,” Advances in Neural Information Processing Systems , vol. 36, pp. 51 991–52 008, 2023

Then

H. Wen, Y. Li, G. Liu, S. Zhao, T. Yu, T. J.-J. Li, S. Jiang, Y. Liu, Y. Zhang, and Y. Liu, “Autodroid: Llm-powered task automation in android,” in Proceedings of the 30th Annual International Conference on Mobile Computing and Networking , 2024, pp. 543–557
2024
Later among the works it cites.
Y. Ge, W. Hua, K. Mei, J. Tan, S. Xu, Z. Li, Y. Zhang et al. , “Openagi: When llm meets domain experts,” Advances in Neural Information Processing Systems , vol. 36, 2024
2024
Later among the works it cites.
Y. Shen, K. Song, X. Tan, D. Li, W. Lu, and Y. Zhuang, “Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face,” Advances in Neural Information Processing Systems , vol. 36, 2024
2024
Later among the works it cites.
T. Guo, X. Chen, Y. Wang, R. Chang, S. Pei, N. Chawla, O. Wiest, and X. Zhang, “Large language model based multi-agents: A survey of progress and challenges.” in 33rd International Joint Conference on Artificial Intelligence (IJCAI 2024) . IJCAI; Cornell arxiv, 2024
2024

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…