The Obvious Invisible Threat: LLM-Powered GUI Agents' Vulnerability to Fine-Print Injections

The theory of affordances:(1979)

James J Gibson. 2014 · 1979

Earlier work this paper cites.

Cognitive load during problem solving: Effects on learning

John Sweller. 1988 · 1988

Earlier work this paper cites.

In two minds: dual-process accounts of reasoning

Jonathan St BT Evans. 2003 · 2003

Earlier work this paper cites.

Visual attention: the where, what, how and why of saliency

Stefan Treue. 2003 · 2003

Earlier work this paper cites.

Privacy as contextual integrity

Helen Nissenbaum. 2004 · 2004

Earlier work this paper cites.

Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among raters

Kilem L Gwet. 2014 · 2014

Earlier work this paper cites.

Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP) . IEEE, 3–18

Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. 2017 · 2017

Earlier work this paper cites.

The dark (patterns) side of UX design. In Proceedings of the 2018 CHI conference on human factors in computing systems . 1–14

Colin M Gray, Yubo Kou, Bryan Battles, Joseph Hoggatt, and Austin L Toombs. 2018 · 2018

Earlier work this paper cites.

Extracting training data from large language models. In 30th USENIX security symposium (USENIX Security 21) . 2633–2650

Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al · 2021

Earlier work this paper cites.

Extracting training data from diffusion models. In 32nd USENIX Security Symposium (USENIX Security 23) . 5253–5270

Nicolas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramer, Borja Balle, Daphne Ippolito, and Eric Wallace. 2023 · 2023

Earlier work this paper cites.

Mind2web: Towards a generalist agent for the web

Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. 2023 · 2023

Cited alongside, same era.

Assistgui: Task-oriented desktop graphical user interface automation

Original

Difei Gao, Lei Ji, Zechen Bai, Mingyu Ouyang, Peiran Li, Dongxing Mao, Qinchen Wu, Weichen Zhang, Peiyi Wang, Xiangwu Guo, et al · 2023

Cited alongside, same era.

Swe-bench: Can language models resolve real-world github issues?

Original

Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023 · 2023

Cited alongside, same era.

Agentbench: Evaluating llms as agents

Original

Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al · 2023

Cited alongside, same era.

Can llms keep a secret? testing privacy implications of language models via contextual integrity theory

Eia: Environmental injection attack on generalist web agents for privacy leakage

Original

Zeyi Liao, Lingbo Mo, Chejian Xu, Mintong Kang, Jiawei Zhang, Chaowei Xiao, Yuan Tian, Bo Li, and Huan Sun. 2024 · 2024

Later among the works it cites.

GUI Agents: A Survey

Original

Dang Nguyen, Jian Chen, Yu Wang, Gang Wu, Namyong Park, Zhengmian Hu, Hanjia Lyu, Junda Wu, Ryan Aponte, Yu Xia, Xintong Li, Jing Shi, Hongjie Chen, Viet Dac Lai, Zhouhang Xie, Sungchul Kim, Ruiyi Zhang, Tong Yu, Mehrab Tanjim, Nesreen K. Ahmed, Puneet Mathur, Seunghyun Yoon, Lina Yao, Branislav Kveton, Thien Huu Nguyen, Trung Bui, Tianyi Zhou, Ryan A. Rossi, and Franck Dernoncourt. 2024 · 2024

Later among the works it cites.

Privacylens: Evaluating privacy norm awareness of language models in action

Original

Yijia Shao, Tianshi Li, Weiyan Shi, Yanchen Liu, and Diyi Yang. 2024 · 2024

Later among the works it cites.

Security matrix for multimodal agents on mobile devices: A systematic and proof of concept study

Original

Yulong Yang, Xinshan Yang, Shuaidong Li, Chenhao Lin, Zhengyu Zhao, Chao Shen, and Tianwei Zhang. 2024 · 2024