MobA: Multifaceted Memory-Enhanced Adaptive Planning for Efficient Mobile Task Automation

Language models are few-shot learners

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020 · 1901

Earlier work this paper cites.

Sugilite: Creating multimodal smartphone automation by demonstration

Toby Jia-Jun Li, Amos Azaria, and Brad A. Myers. 2017 · 2017

Earlier work this paper cites.

Pumice: A multi-modal agent that learns concepts and conditionals from natural language and demonstrations

Toby Jia-Jun Li, Marissa Radensky, Justin Jia, Kirielle Singarajah, Tom M Mitchell, and Brad A Myers. 2019 · 2019

Earlier work this paper cites.

Uibert: Learning generic multimodal representations for ui understanding

Original

Chongyang Bai, Xiaoxue Zang, Ying Xu, Srinivas Sunkara, Abhinav Rastogi, Jindong Chen, and Blaise Aguera y Arcas. 2021 · 2021

Earlier work this paper cites.

Actionbert: Leveraging user actions for semantic understanding of user interfaces

Zecheng He, Srinivas Sunkara, Xiaoxue Zang, Ying Xu, Lijuan Liu, Nevan Wichers, Gabriel Schubiner, Ruby Lee, and Jindong Chen. 2021 · 2021

Earlier work this paper cites.

Screen2vec: Semantic embedding of gui screens and gui components

Toby Jia-Jun Li, Lindsay Popowski, Tom Mitchell, and Brad A Myers. 2021a · 2021

Earlier work this paper cites.

Learning transferable visual models from natural language supervision

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021 · 2021

Earlier work this paper cites.

Screen2words: Automatic mobile ui summarization with multimodal learning

Bryan Wang, Gang Li, Xin Zhou, Zhourong Chen, Tovi Grossman, and Yang Li. 2021 · 2021

Earlier work this paper cites.

Rethinking the role of demonstrations: What makes in-context learning work?

Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022 · 2022

Earlier work this paper cites.

META-GUI: Towards multi-modal conversational agents on mobile GUI

Liangtai Sun, Xingyu Chen, Lu Chen, Tianle Dai, Zichen Zhu, and Kai Yu. 2022 · 2022

Earlier work this paper cites.

Chain-of-thought prompting elicits reasoning in large language models

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022 · 2022

Earlier work this paper cites.

Using large language models to simulate multiple humans and replicate human subject studies

Gati Aher, Rosa I. Arriaga, and Adam Tauman Kalai. 2023 · 2023

Earlier work this paper cites.

Minigpt-v2: large language model as a unified interface for vision-language multi-task learning

Original

Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. 2023 · 2023

Earlier work this paper cites.

Instructblip: Towards general-purpose vision-language models with instruction tuning

Wenliang Dai, Junnan Li, DONGXU LI, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. 2023 · 2023

Earlier work this paper cites.

Understanding the benefits and challenges of deploying conversational ai leveraging large language models for public health intervention

Eunkyung Jo, Daniel A. Epstein, Hyunhoon Jung, and Young-Ho Kim. 2023 · 2023

Earlier work this paper cites.

Spotlight: Mobile ui understanding using vision-language models with a focus

Gang Li and Yang Li. 2023 · 2023

Earlier work this paper cites.

Tradinggpt: Multi-agent system with layered memory and distinct characters for enhanced financial trading performance

Original

Yang Li, Yangyang Yu, Haohang Li, Zhi Chen, and Khaldoun Khashanah. 2023 · 2023

Earlier work this paper cites.

Visual instruction tuning

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023 · 2023

Earlier work this paper cites.

Self-refine: Iterative refinement with self-feedback

Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2023 · 2023

Cited alongside, same era.

Gpt-4v(ision) system card

OpenAI. 2023 · 2023

Cited alongside, same era.

Generative agents: Interactive simulacra of human behavior

Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2023 · 2023

Cited alongside, same era.

Reflexion: language agents with verbal reinforcement learning

Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023 · 2023

Cited alongside, same era.

Gemini in reasoning: Unveiling commonsense in multimodal large language models

Original

Yuqing Wang and Yun Zhao. 2023 · 2023

Cited alongside, same era.

Understanding the planning of llm agents: A survey

Original

Xu Huang, Weiwen Liu, Xiaolong Chen, Xingmei Wang, Hao Wang, Defu Lian, Yasheng Wang, Ruiming Tang, and Enhong Chen. 2024 · 2024

Closest in time.

Depression diagnosis dialogue simulation: Self-improving psychiatrist with tertiary memory

Original

Kunyao Lan, Bingui Jin, Zichen Zhu, Siyuan Chen, Shu Zhang, Kenny Q. Zhu, and Mengyue Wu. 2024 · 2024

Closest in time.

CoCo-agent: A comprehensive cognitive MLLM agent for smartphone GUI automation

Xinbei Ma, Zhuosheng Zhang, and Hai Zhao. 2024 · 2024

Closest in time.

Evaluating very long-term conversational memory of llm agents

Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, and Yuwei Fang. 2024 · 2024

Closest in time.

Mobileflow: A multimodal llm for mobile gui agent

Original

Songqin Nong, Jiali Zhu, Rui Wu, Jiongchao Jin, Shuo Shan, Xiutian Huang, and Wenhao Xu. 2024 · 2024