MIP against Agent: Malicious Image Patches Hijacking Multimodal OS Agents

An overview of the Tesseract OCR engine

Ray Smith · 2007

Earlier work this paper cites.

Intriguing properties of neural networks

Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus · 2014

Earlier work this paper cites.

Explaining and harnessing adversarial examples

Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy · 2015

Earlier work this paper cites.

Adam: A method for stochastic optimization

Diederik P. Kingma and Jimmy Ba · 2015

Earlier work this paper cites.

Adversarial examples in the physical world

Original

Alexey Kurakin, Ian Goodfellow, and Samy Bengio · 2016

Earlier work this paper cites.

Adversarial examples are not easily detected: Bypassing ten detection methods

Nicholas Carlini and David A. Wagner · 2017

Earlier work this paper cites.

Attention is all you need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin · 2017

Earlier work this paper cites.

Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples

Anish Athalye, Nicholas Carlini, and David Wagner · 2018

Earlier work this paper cites.

Towards deep learning models resistant to adversarial attacks

Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu · 2018

Earlier work this paper cites.

Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks

Francesco Croce and Matthias Hein · 2020

Earlier work this paper cites.

Improving image captioning with better use of captions

Original

Zhan Shi, Xu Zhou, Xipeng Qiu, and Xiaodan Zhu · 2020

Earlier work this paper cites.

Invisible for both camera and LiDAR: Security of multi-sensor fusion based perception in autonomous driving under physical-world attacks

Yulong Cao, Ningfei Wang, Chaowei Xiao, Dawei Yang, Jin Fang, Ruigang Yang, Qi Alfred Chen, Mingyan Liu, and Bo Li · 2021

Earlier work this paper cites.

A survey on adversarial attacks and defences

Anirban Chakraborty, Manaar Alam, Vishal Dey, Anupam Chattopadhyay, and Debdeep Mukhopadhyay · 2021

Earlier work this paper cites.

Training language models to follow instructions with human feedback

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe · 2022

Earlier work this paper cites.

Towards adversarial attack on vision-language pre-training models

Jiaming Zhang, Qi Yi, and Jitao Sang · 2022

Earlier work this paper cites.

Gemini: A family of highly capable multimodal models

Original

Gemini Team · 2023

Earlier work this paper cites.

SA-Attack: Improving adversarial transferability of vision-language pre-training models via self-augmentation

Original

Bangyan He, Xiaojun Jia, Siyuan Liang, Tianrui Lou, Yang Liu, and Xiaochun Cao · 2023

Earlier work this paper cites.

A reading survey on adversarial machine learning: Adversarial attacks and their understanding

Original

Shashank Kotyan · 2023

Earlier work this paper cites.

Set-level guidance attack: Boosting adversarial transferability of vision-language pre-training models

Dong Lu, Zhiqiang Wang, Teng Wang, Weili Guan, Hongchang Gao, and Feng Zheng · 2023

Earlier work this paper cites.

gpt-engineer, 2023

Anton Osika et al · 2023

Earlier work this paper cites.

Calibrated interpretation: Confidence estimation in semantic parsing

Elias Stengel-Eskin and Benjamin Van Durme · 2023

Cited alongside, same era.

Set-of-Mark prompting unleashes extraordinary visual grounding in GPT-4V

Original

Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao · 2023

Cited alongside, same era.

On evaluating adversarial robustness of large vision-language models

Yunqing Zhao, Tianyu Pang, Chao Du, Xiao Yang, Chongxuan Li, Ngai-Man Cheung, and Min Lin · 2023

Cited alongside, same era.

Universal and transferable adversarial attacks on aligned language models

Original

Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson · 2023

Cited alongside, same era.

Image hijacks: Adversarial images can control generative models at runtime

Luke Bailey, Euan Ong, Stuart Russell, and Scott Emmons · 2024

Cited alongside, same era.

Robust CLIP: Unsupervised adversarial fine-tuning of vision embeddings for robust large vision-language models

Christian Schlarmann, Naman Deep Singh, Francesco Croce, and Matthias Hein · 2024

Later among the works it cites.

Jailbreak defense in a narrow domain: Limitations of existing methods and a new transcript-classifier approach

Original

Tony T. Wang, John Hughes, Henry Sleight, Rylan Schaeffer, Rajashree Agrawal, Fazl Barez, Mrinank Sharma, Jesse Mu, Nir Shavit, and Ethan Perez · 2024

Later among the works it cites.

OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments

Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu · 2024

Later among the works it cites.

Enhancing transferable adversarial attacks on vision transformers through gradient normalization scaling and high-frequency adaptation

Zhiyu Zhu, Xinyi Wang, Zhibo Jin, Jiayu Zhang, and Huaming Chen · 2024

Later among the works it cites.