Caution for the Environment: Multimodal LLM Agents are Susceptible to Environmental Distractions

The probabilistic relevance framework: Bm25 and beyond

Stephen Robertson, Hugo Zaragoza, et al. 2009 · 2009

Earlier work this paper cites.

LoRA: Low-rank adaptation of large language models

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022 · 2022

Earlier work this paper cites.

Training language models to follow instructions with human feedback

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022 · 2022

Earlier work this paper cites.

META-GUI: Towards multi-modal conversational agents on mobile GUI

Liangtai Sun, Xingyu Chen, Lu Chen, Tianle Dai, Zichen Zhu, and Kai Yu. 2022 · 2022

Earlier work this paper cites.

Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond

Original

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023 · 2023

Earlier work this paper cites.

Cogagent: A visual language model for gui agents

Original

Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. 2023 · 2023

Earlier work this paper cites.

Evaluating the instruction-following robustness of large language models to prompt injection

Zekun Li, Baolin Peng, Pengcheng He, and Xifeng Yan. 2023 · 2023

Earlier work this paper cites.

Visual instruction tuning

Original

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023 · 2023

Earlier work this paper cites.

When not to trust language models: Investigating effectiveness of parametric and non-parametric memories

Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023 · 2023

Earlier work this paper cites.

Gpt-4 technical report

Original

OpenAI. 2023 · 2023

Earlier work this paper cites.

Instruction tuning with gpt-4

Original

Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. 2023 · 2023

Earlier work this paper cites.

Androidinthewild: A large-scale dataset for android device control

Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy P Lillicrap. 2023 · 2023

Earlier work this paper cites.

Large language models can be easily distracted by irrelevant context

Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H Chi, Nathanael Schärli, and Denny Zhou. 2023 · 2023

Earlier work this paper cites.

Powerinfer: Fast large language model serving with a consumer-grade gpu

Original

Yixin Song, Zeyu Mi, Haotong Xie, and Haibo Chen. 2023 · 2023

Earlier work this paper cites.

Evil geniuses: Delving into the safety of llm-based agents

Original

Yu Tian, Xiao Yang, Jingyuan Zhang, Yinpeng Dong, and Hang Su. 2023 · 2023

Cited alongside, same era.

Describe, explain, plan and select: interactive planning with large language models enables open-world multi-task agents

Zihao Wang, Shaofei Cai, Guanzhou Chen, Anji Liu, Xiaojian Ma, Yitao Liang, and Team CraftJarvis. 2023 · 2023

Cited alongside, same era.

Appagent: Multimodal agents as smartphone users

Original

Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. 2023 · 2023

Cited alongside, same era.

You only look at screens: Multimodal chain-of-action agents

Original

Zhuosheng Zhang and Aston Zhang. 2023 · 2023

Cited alongside, same era.

Memorybank: Enhancing large language models with long-term memory

SoFA: Shielded on-the-fly alignment via priority rule following

Xinyu Lu, Bowen Yu, Yaojie Lu, Hongyu Lin, Haiyang Yu, Le Sun, Xianpei Han, and Yongbin Li. 2024 · 2024

Closest in time.

Comprehensive cognitive llm agent for smartphone gui automation

Xinbei Ma, Zhuosheng Zhang, and Hai Zhao. 2024 · 2024

Closest in time.

Direct preference optimization: Your language model is secretly a reward model

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024 · 2024

Closest in time.

Identifying the risks of LM agents with an LM-emulated sandbox

Yangjun Ruan, Honghua Dong, Andrew Wang, Silviu Pitis, Yongchao Zhou, Jimmy Ba, Yann Dubois, Chris J. Maddison, and Tatsunori Hashimoto. 2024 · 2024

Closest in time.

Reflexion: Language agents with verbal reinforcement learning

Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2024 · 2024

Closest in time.