MLA-Trust: Benchmarking Trustworthiness of Multimodal LLM Agents in GUI Environments

W. Johannsen, “Elemente der exakten erblichkeitslehre. 1909,” Gustav Fischer, Jena , 1913

1913

Earlier work this paper cites.

X. Zhao, M. Li, C. Weber, M. B. Hafez, and S. Wermter, “Chat with the environment: Interactive multimodal perception using large language models,” arXiv preprint arXiv:2303.08268 , 2023

Original

2023

Earlier work this paper cites.

S. Zheng, J. Liu, Y. Feng, and Z. Lu, “Steve-eye: Equipping llm-based embodied agents with visual perception in open worlds,” arXiv preprint arXiv:2310.13255 , 2023

Original

2023

Earlier work this paper cites.

C. Zhang, Z. Yang, J. Liu, Y. Han, X. Chen, Z. Huang, B. Fu, and G. Yu, “Appagent: Multimodal agents as smartphone users,” arXiv preprint arXiv:2312.13771 , 2023

Original

2023

Earlier work this paper cites.

Z. Zhang and A. Zhang, “You only look at screens: Multimodal chain-of-action agents,” arXiv preprint arXiv:2309.10143 , 2023

Original

2023

Earlier work this paper cites.

Y. Ruan, H. Dong, A. Wang, S. Pitis, Y. Zhou, J. Ba, Y. Dubois, C. J. Maddison, and T. Hashimoto, “Identifying the risks of lm agents with an lm-emulated sandbox,” arXiv preprint arXiv:2309.15817 , 2023

Original

2023

Earlier work this paper cites.

H. Tu, C. Cui, Z. Wang, Y. Zhou, B. Zhao, J. Han, W. Zhou, H. Yao, and C. Xie, “How many unicorns are in this image? a safety evaluation benchmark for vision llms,” arXiv preprint arXiv:2311.16101 , 2023

Original

2023

Earlier work this paper cites.

Z. Xi, W. Chen, and X. Guo, “The rise and potential of large language model based agents: A survey,” arXiv preprint arXiv:2309.07864 , 2023

Original

2023

Earlier work this paper cites.

T. Sumers, S. Yao, K. Narasimhan, and T. Griffiths, “Cognitive architectures for language agents,” Transactions on Machine Learning Research , 2023

2023

Earlier work this paper cites.

D. Surís, S. Menon, and C. Vondrick, “Vipergpt: Visual inference via python execution for reasoning,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 11 888–11 898

2023

Earlier work this paper cites.

P. Lu, B. Peng, H. Cheng, M. Galley, K.-W. Chang, Y. N. Wu, S.-C. Zhu, and J. Gao, “Chameleon: Plug-and-play compositional reasoning with large language models,” Advances in Neural Information Processing Systems , vol. 36, pp. 43 447–43 478, 2023

2023

Earlier work this paper cites.

Y. Shen, K. Song, X. Tan, D. Li, W. Lu, and Y. Zhuang, “Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face,” Advances in Neural Information Processing Systems , vol. 36, pp. 38 154–38 180, 2023

2023

Earlier work this paper cites.

Z. Yang, L. Li, J. Wang, K. Lin, E. Azarnasab, F. Ahmed, Z. Liu, C. Liu, M. Zeng, and L. Wang, “Mm-react: Prompting chatgpt for multimodal reasoning and action,” arXiv preprint arXiv:2303.11381 , 2023

Original

2023

Earlier work this paper cites.

J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023

Original

2023

Earlier work this paper cites.

J. Betker, G. Goh, L. Jing, T. Brooks, J. Wang, L. Li, L. Ouyang, J. Zhuang, J. Lee, Y. Guo et al. , “Improving image generation with better captions,” Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf , vol. 2, no. 3, p. 8, 2023

2023

Earlier work this paper cites.

C. Wu, S. Yin, W. Qi, X. Wang, Z. Tang, and N. Duan, “Visual chatgpt: Talking, drawing and editing with visual foundation models,” arXiv preprint arXiv:2303.04671 , 2023

Original

2023

Earlier work this paper cites.

A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford et al. , “Gpt-4o system card,” arXiv preprint arXiv:2410.21276 , 2024

Original

2024

Earlier work this paper cites.

DeepMind, “Gemini 2.0 flash model card,” Google, Tech. Rep., 2024

2024

Earlier work this paper cites.

Anthropic, “The claude 3 model family: Opus, sonnet, haiku,” 2024. [Online]. Available: https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf

2024

Earlier work this paper cites.

P. Agrawal, S. Antoniak, E. B. Hanna, B. Bout, D. Chaplot, J. Chudnovsky, D. Costa, B. De Monicault, S. Garg, T. Gervet et al. , “Pixtral 12b,” arXiv preprint arXiv:2410.07073 , 2024

Original

2024

Earlier work this paper cites.

X. Liu, T. Zhang, Y. Gu, I. L. Iong, Y. Xu, X. Song, S. Zhang, H. Lai, X. Liu, H. Zhao et al. , “Visualagentbench: Towards large multimodal models as visual foundation agents,” arXiv preprint arXiv:2408.06327 , 2024

Original

2024

Earlier work this paper cites.

J. Xie, Z. Chen, R. Zhang, X. Wan, and G. Li, “Large multimodal agents: A survey,” arXiv preprint arXiv:2402.15116 , 2024

Original

2024

Earlier work this paper cites.

Z. Durante, Q. Huang, N. Wake, R. Gong, J. S. Park, B. Sarkar, R. Taori, Y. Noda, D. Terzopoulos, Y. Choi et al. , “Agent ai: Surveying the horizons of multimodal interaction,” arXiv preprint arXiv:2401.03568 , 2024

Original

2024

Earlier work this paper cites.

H. He, W. Yao, K. Ma, W. Yu, Y. Dai, H. Zhang, Z. Lan, and D. Yu, “Webvoyager: Building an end-to-end web agent with large multimodal models,” arXiv preprint arXiv:2401.13919 , 2024

Original

2024

Cited alongside, same era.

J. Wang, H. Xu, J. Ye, M. Yan, W. Shen, J. Zhang, F. Huang, and J. Sang, “Mobile-agent: Autonomous multi-modal mobile device agent with visual perception,” arXiv preprint arXiv:2401.16158 , 2024

Original

2024

Cited alongside, same era.

J. Wang, H. Xu, H. Jia, X. Zhang, M. Yan, W. Shen, J. Zhang, F. Huang, and J. Sang, “Mobile-agent-v2: Mobile device operation assistant with effective navigation via multi-agent collaboration,” arXiv preprint arXiv:2406.01014 , 2024

Original

2024

Cited alongside, same era.

J. Y. Koh, R. Lo, L. Jang, V. Duvvur, M. C. Lim, P.-Y. Huang, G. Neubig, S. Zhou, R. Salakhutdinov, and D. Fried, “Visualwebarena: Evaluating multimodal agents on realistic visual web tasks,” arXiv preprint arXiv:2401.13649 , 2024

Original

2024

Cited alongside, same era.

H. Zhang, J. Huang, K. Mei, Y. Yao, Z. Wang, C. Zhan, H. Wang, and Y. Zhang, “Agent security bench (asb): Formalizing and benchmarking attacks and defenses in llm-based agents,” arXiv preprint arXiv:2410.02644 , 2024

Original

2024

Later among the works it cites.

I. Levy, B. Wiesel, S. Marreed, A. Oved, A. Yaeli, and S. Shlomov, “St-webagentbench: A benchmark for evaluating safety and trustworthiness in web agents,” arXiv preprint arXiv:2410.06703 , 2024

Original

2024

Later among the works it cites.

Z. Zhang, S. Cui, Y. Lu, J. Zhou, J. Yang, H. Wang, and M. Huang, “Agent-safetybench: Evaluating the safety of llm agents,” arXiv preprint arXiv:2412.14470 , 2024

Original

2024

Later among the works it cites.

S. Liu, J. Chen, S. Ruan, H. Su, and Z. Yin, “Exploring the robustness of decision-level through adversarial attacks on llm-based embodied models,” in Proceedings of the 32nd ACM International Conference on Multimedia , 2024, pp. 8120–8128

2024

Later among the works it cites.