Built on

S. Gu and L. Rigazio, “Towards Deep Neural Network Architectures Robust to Adversarial Examples,” in International Conference on Learning Representations (ICLR) , 2015
2015
Earlier work this paper cites.
D. Meng and H. Chen, “MagNet: A Two-Pronged Defense against Adversarial Examples,” in Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, CCS 2017, Dallas, TX, USA, October 30 - November 03, 2017 . ACM, 2017, pp. 135–147
2017
Earlier work this paper cites.
J. H. Metzen, T. Genewein, V. Fischer, and B. Bischoff, “On Detecting Adversarial Perturbations,” in International Conference on Learning Representations (ICLR) , 2017
2017
Earlier work this paper cites.
K. Grosse, P. Manoharan, N. Papernot, M. Backes, and P. D. McDaniel, “On the (Statistical) Detection of Adversarial Examples,” Computing Research Repository (CoRR) , vol. abs/1702.06280, 2017
Original
2017
Earlier work this paper cites.
A. N. Bhagoji, D. Cullina, C. Sitawarin, and P. Mittal, “Enhancing robustness of machine learning systems via data transformations,” in 52nd Annual Conference on Information Sciences and Systems, CISS 2018, Princeton, NJ, USA, March 21-23, 2018 . IEEE, 2018, pp. 1–5
2018
Earlier work this paper cites.
P. Samangouei, M. Kabkab, and R. Chellappa, “Defense-GAN: Protecting Classifiers Against Adversarial Attacks Using Generative Models,” in International Conference on Learning Representations (ICLR) , 2018
2018
Earlier work this paper cites.
A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards deep learning models resistant to adversarial attacks,” corr , no. arXiv:1706.06083, 2019. [Online]. Available: http://arxiv.org/abs/1706.06083
Original
2019
Earlier work this paper cites.
C. Zhu, Y. Cheng, Z. Gan, S. Sun, and T. G. et al., “FreeLB: Enhanced Adversarial Training for Natural Language Understanding,” in International Conference on Learning Representations (ICLR) , 2020
2020
Earlier work this paper cites.
N. Carlini, F. Tramèr, E. Wallace, M. Jagielski, and A. H. et al., “Extracting Training Data from Large Language Models,” in USENIX Security Symposium , 2021
2021
Earlier work this paper cites.
L. Weidinger, J. Mellor, M. Rauh, C. Griffin, and J. U. et al., “Ethical and social risks of harm from Language Models,” Computing Research Repository (CoRR) , vol. abs/2112.04359, 2021
Original
2021
Earlier work this paper cites.
R. Bommasani, D. A. Hudson, E. Adeli, and R. Altman, “On the opportunities and risks of foundation models,” corr , no. arXiv:2108.07258, 2022. [Online]. Available: http://arxiv.org/abs/2108.07258
Original
2022
Earlier work this paper cites.
H. Chase. (2022) LangChain. Langchain. Original-date: 2022-10-17T02:58:36Z. [Online]. Available: https://github.com/langchain-ai/langchain
2022
Earlier work this paper cites.
W. Nie, B. Guo, Y. Huang, C. Xiao, and A. V. et al., “Diffusion Models for Adversarial Purification,” in International Conference on Machine Learning (ICML) , ser. Proceedings of Machine Learning Research, vol. 162. PMLR, 2022, pp. 16 805–16 827
2022
Earlier work this paper cites.
E. Derner and K. Batistič, “Beyond the Safeguards: Exploring the Security Risks of ChatGPT,” Computing Research Repository (CoRR) , vol. abs/2305.08005, 2023
Original
2023
Earlier work this paper cites.
G. Deng, Y. Liu, Y. Li, K. Wang, and Y. Z. et al., “Jailbreaker: Automated Jailbreak Across Multiple Large Language Model Chatbots,” Computing Research Repository (CoRR) , vol. abs/2307.08715, 2023
Original
2023
Earlier work this paper cites.
S. Zhu, R. Zhang, B. An, G. Wu, and J. B. et al., “AutoDAN: Automatic and Interpretable Adversarial Attacks on Large Language Models,” Computing Research Repository (CoRR) , vol. abs/2310.15140, 2023
Original
2023
Earlier work this paper cites.
Y. Liu, Y. Jia, R. Geng, J. Jia, and N. Z. Gong, “Prompt Injection Attacks and Defenses in LLM-Integrated Applications,” Computing Research Repository (CoRR) , vol. abs/2310.12815, 2023
Original
2023
Earlier work this paper cites.
E. Crothers, N. Japkowicz, and H. L. Viktor, “Machine-generated text: A comprehensive survey of threat models and detection methods,” IEEE Access , 2023
2023
Earlier work this paper cites.
S. Yao, J. Zhao, D. Yu, N. Du, and I. e. a. Shafran, “ReAct: Synergizing reasoning and acting in language models,” corr , no. arXiv:2210.03629, 2023. [Online]. Available: http://arxiv.org/abs/2210.03629
Original
2023
Earlier work this paper cites.
K. Greshake, S. Abdelnabi, S. Mishra, C. Endres, T. Holz, and M. Fritz, “Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injection,” in Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security, AISec 2023, Copenhagen, Denmark, 30 November 2023 . ACM, 2023, pp. 79–90. [Online]. Available: https://doi.org/10.1145/3605764.3623985
2023
Earlier work this paper cites.

Similar

D. Kang, X. Li, I. Stoica, C. Guestrin, and M. Z. et al., “Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks,” Computing Research Repository (CoRR) , vol. abs/2302.05733, 2023
Original
2023
Cited alongside, same era.
M. Shergadwala. (2023) Prompt injection attacks in various LLMs. [Online]. Available: https://medium.com/@murtuza.shergadwala/prompt-injection-attacks-in-various-llms-206f56cd6ee9
2023
Cited alongside, same era.
A. Wei, N. Haghtalab, and J. Steinhardt, “Jailbroken: How Does LLM Safety Training Fail?” in Annual Conference on Neural Information Processing Systems (NeurIPS) , 2023
2023
Cited alongside, same era.
A. Zou, Z. Wang, J. Z. Kolter, and M. Fredrikson, “Universal and Transferable Adversarial Attacks on Aligned Language Models,” Computing Research Repository (CoRR) , vol. abs/2307.15043, 2023
Original
2023
Cited alongside, same era.

Then

J. Kirchenbauer, J. Geiping, Y. Wen, M. Shu, and K. S. et al., “On the Reliability of Watermarks for Large Language Models,” Computing Research Repository (CoRR) , vol. abs/2306.04634, 2023
Original
2023
Later among the works it cites.
A. Robey, E. Wong, H. Hassani, and G. J. Pappas, “Smoothllm: Defending large language models against jailbreaking attacks,” Computing Research Repository (CoRR) , vol. abs/2310.03684, 2023
Original
2023
Later among the works it cites.
Notion. (2024) Notion AI | now with q&a. [Online]. Available: https://www.notion.so/product/ai
2024
Closest in time.
OpenAI. (2024) Integrate the OpenAI (ChatGPT) API with the gmail API. OpenAI. [Online]. Available: https://pipedream.com/apps/openai/integrations/gmail
2024
Closest in time.
Clockwise. (2024) AI calendar | AI scheduling assistant | clockwise. Clockwise. [Online]. Available: https://www.getclockwise.com/ai

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2024

Whispers in the Machine: Confidentiality in Agentic Systems

Evertz, Jonathan, Chlosta, Merlin, Schönherr, Lea et al.

Understand

Large language model (LLM)-based agents combine LLMs with external tools to automate tasks such as scheduling meetings, managing documents, or booking travel.

While these integrations unlock powerful capabilities, they also create new and more severe attack surfaces.
In particular, prompt injection attacks become far more dangerous in the agentic setting: malicious instructions embedded in connected services can misdirect the agent, providing a direct pathway for sensitive data to be exfiltrated.
Yet, despite a growing number of real-world incidents, the confidentiality risks of such systems remain poorly understood.

Built on

S. Gu and L. Rigazio, “Towards Deep Neural Network Architectures Robust to Adversarial Examples,” in International Conference on Learning Representations (ICLR) , 2015
2015
Earlier work this paper cites.
D. Meng and H. Chen, “MagNet: A Two-Pronged Defense against Adversarial Examples,” in Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, CCS 2017, Dallas, TX, USA, October 30 - November 03, 2017 . ACM, 2017, pp. 135–147
2017
Earlier work this paper cites.
J. H. Metzen, T. Genewein, V. Fischer, and B. Bischoff, “On Detecting Adversarial Perturbations,” in International Conference on Learning Representations (ICLR) , 2017
2017
Earlier work this paper cites.
K. Grosse, P. Manoharan, N. Papernot, M. Backes, and P. D. McDaniel, “On the (Statistical) Detection of Adversarial Examples,” Computing Research Repository (CoRR) , vol. abs/1702.06280, 2017
Original
2017
Earlier work this paper cites.
A. N. Bhagoji, D. Cullina, C. Sitawarin, and P. Mittal, “Enhancing robustness of machine learning systems via data transformations,” in 52nd Annual Conference on Information Sciences and Systems, CISS 2018, Princeton, NJ, USA, March 21-23, 2018 . IEEE, 2018, pp. 1–5
2018
Earlier work this paper cites.
P. Samangouei, M. Kabkab, and R. Chellappa, “Defense-GAN: Protecting Classifiers Against Adversarial Attacks Using Generative Models,” in International Conference on Learning Representations (ICLR) , 2018
2018
Earlier work this paper cites.
A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards deep learning models resistant to adversarial attacks,” corr , no. arXiv:1706.06083, 2019. [Online]. Available: http://arxiv.org/abs/1706.06083
Original
2019
Earlier work this paper cites.
C. Zhu, Y. Cheng, Z. Gan, S. Sun, and T. G. et al., “FreeLB: Enhanced Adversarial Training for Natural Language Understanding,” in International Conference on Learning Representations (ICLR) , 2020
2020
Earlier work this paper cites.
N. Carlini, F. Tramèr, E. Wallace, M. Jagielski, and A. H. et al., “Extracting Training Data from Large Language Models,” in USENIX Security Symposium , 2021
2021
Earlier work this paper cites.
L. Weidinger, J. Mellor, M. Rauh, C. Griffin, and J. U. et al., “Ethical and social risks of harm from Language Models,” Computing Research Repository (CoRR) , vol. abs/2112.04359, 2021
Original
2021
Earlier work this paper cites.
R. Bommasani, D. A. Hudson, E. Adeli, and R. Altman, “On the opportunities and risks of foundation models,” corr , no. arXiv:2108.07258, 2022. [Online]. Available: http://arxiv.org/abs/2108.07258
Original
2022
Earlier work this paper cites.
H. Chase. (2022) LangChain. Langchain. Original-date: 2022-10-17T02:58:36Z. [Online]. Available: https://github.com/langchain-ai/langchain
2022
Earlier work this paper cites.
W. Nie, B. Guo, Y. Huang, C. Xiao, and A. V. et al., “Diffusion Models for Adversarial Purification,” in International Conference on Machine Learning (ICML) , ser. Proceedings of Machine Learning Research, vol. 162. PMLR, 2022, pp. 16 805–16 827
2022
Earlier work this paper cites.
E. Derner and K. Batistič, “Beyond the Safeguards: Exploring the Security Risks of ChatGPT,” Computing Research Repository (CoRR) , vol. abs/2305.08005, 2023
Original
2023
Earlier work this paper cites.
G. Deng, Y. Liu, Y. Li, K. Wang, and Y. Z. et al., “Jailbreaker: Automated Jailbreak Across Multiple Large Language Model Chatbots,” Computing Research Repository (CoRR) , vol. abs/2307.08715, 2023
Original
2023
Earlier work this paper cites.
S. Zhu, R. Zhang, B. An, G. Wu, and J. B. et al., “AutoDAN: Automatic and Interpretable Adversarial Attacks on Large Language Models,” Computing Research Repository (CoRR) , vol. abs/2310.15140, 2023
Original
2023
Earlier work this paper cites.
Y. Liu, Y. Jia, R. Geng, J. Jia, and N. Z. Gong, “Prompt Injection Attacks and Defenses in LLM-Integrated Applications,” Computing Research Repository (CoRR) , vol. abs/2310.12815, 2023
Original
2023
Earlier work this paper cites.
E. Crothers, N. Japkowicz, and H. L. Viktor, “Machine-generated text: A comprehensive survey of threat models and detection methods,” IEEE Access , 2023
2023
Earlier work this paper cites.
S. Yao, J. Zhao, D. Yu, N. Du, and I. e. a. Shafran, “ReAct: Synergizing reasoning and acting in language models,” corr , no. arXiv:2210.03629, 2023. [Online]. Available: http://arxiv.org/abs/2210.03629
Original
2023
Earlier work this paper cites.
K. Greshake, S. Abdelnabi, S. Mishra, C. Endres, T. Holz, and M. Fritz, “Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injection,” in Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security, AISec 2023, Copenhagen, Denmark, 30 November 2023 . ACM, 2023, pp. 79–90. [Online]. Available: https://doi.org/10.1145/3605764.3623985
2023
Earlier work this paper cites.

Similar

D. Kang, X. Li, I. Stoica, C. Guestrin, and M. Z. et al., “Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks,” Computing Research Repository (CoRR) , vol. abs/2302.05733, 2023
Original
2023
Cited alongside, same era.
M. Shergadwala. (2023) Prompt injection attacks in various LLMs. [Online]. Available: https://medium.com/@murtuza.shergadwala/prompt-injection-attacks-in-various-llms-206f56cd6ee9
2023
Cited alongside, same era.
A. Wei, N. Haghtalab, and J. Steinhardt, “Jailbroken: How Does LLM Safety Training Fail?” in Annual Conference on Neural Information Processing Systems (NeurIPS) , 2023
2023
Cited alongside, same era.
A. Zou, Z. Wang, J. Z. Kolter, and M. Fredrikson, “Universal and Transferable Adversarial Attacks on Aligned Language Models,” Computing Research Repository (CoRR) , vol. abs/2307.15043, 2023
Original
2023
Cited alongside, same era.

Then

J. Kirchenbauer, J. Geiping, Y. Wen, M. Shu, and K. S. et al., “On the Reliability of Watermarks for Large Language Models,” Computing Research Repository (CoRR) , vol. abs/2306.04634, 2023
Original
2023
Later among the works it cites.
A. Robey, E. Wong, H. Hassani, and G. J. Pappas, “Smoothllm: Defending large language models against jailbreaking attacks,” Computing Research Repository (CoRR) , vol. abs/2310.03684, 2023
Original
2023
Later among the works it cites.
Notion. (2024) Notion AI | now with q&a. [Online]. Available: https://www.notion.so/product/ai
2024
Closest in time.
OpenAI. (2024) Integrate the OpenAI (ChatGPT) API with the gmail API. OpenAI. [Online]. Available: https://pipedream.com/apps/openai/integrations/gmail
2024
Closest in time.
Clockwise. (2024) AI calendar | AI scheduling assistant | clockwise. Clockwise. [Online]. Available: https://www.getclockwise.com/ai

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

K. Lee, “0xk1h0 - github: Jailbreak prompts collection,” 2023. [Online]. Available: https://github.com/0xk1h0/ChatGPT_DAN

2023

Cited alongside, same era.

W. Zhang. (2023) Prompt injection attack on GPT-4 — robust intelligence. [Online]. Available: https://www.robustintelligence.com/blog-posts/prompt-injection-attack-on-gpt-4

2023

Cited alongside, same era.

LaurieWired [@lauriewired]. (2023) Novel jailbreak technique via typoglycemia. [Online]. Available: https://twitter.com/lauriewired/status/1682825249203662848

2023

Cited alongside, same era.

K. Zhu, J. Wang, J. Zhou, Z. Wang, and H. C. et al., “PromptBench: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts,” Computing Research Repository (CoRR) , vol. abs/2306.04528, 2023

Original

2023

Cited alongside, same era.

S. Armstrong and R. Gorman, “Using GPT-eliezer against ChatGPT jailbreaking,” 2023. [Online]. Available: https://www.alignmentforum.org/posts/pNcFYZnPdXyL2RfgA/using-gpt-eliezer-against-chatgpt-jailbreaking

2023

Cited alongside, same era.

Y. Liu, Y. Jia, R. Geng, J. Jia, and N. Z. Gong, “Prompt Injection Attacks and Defenses in LLM-Integrated Applications,” Computing Research Repository (CoRR) , vol. abs/2310.12815, 2023

Original

2023

Cited alongside, same era.

LearnPrompting. (2023) Learn prompting. [Online]. Available: https://learnprompting.org/docs/category/-defensive-measures

2023

Cited alongside, same era.

G. Alon and M. Kamfonas, “Detecting Language Model Attacks with Perplexity,” Computing Research Repository (CoRR) , vol. abs/2308.14132, 2023

Original

2023

Cited alongside, same era.

2024

Closest in time.

Microsoft. (2024) Personal AI assistant | microsoft copilot. [Online]. Available: https://www.microsoft.com/en-us/microsoft-copilot/personal-ai-assistant

2024

Closest in time.

J. Rehberger. (2024) Microsoft copilot: From prompt injection to exfiltration of personal information · embrace the red. [Online]. Available: https://embracethered.com/blog/posts/2024/m365-copilot-prompt-injection-tool-invocation-and-data-exfil-using-ascii-smuggling/

2024

Closest in time.

E. Debenedetti, J. Rando, D. Paleka, S. F. Florin, D. Albastroiu, Y. L. Niv Cohen, A. S. Reshmi Ghosh, Rui Wen, S. Z.-B. Giovanni Cherubin, R. Schmid, V. Klem, S. K. Takahiro Miki, Chenhao Li, M. Fritz, F. Tramèr, S. Abdelnabi, and L. Schönherr, “Dataset and lessons learned from the 2024 satml llm capture-the-flag competition,” in NeurIPS Dataset and Benchmark Track , 2024

2024

Closest in time.

A. Dubey, A. Jauhri, A. Pandey, and A. Kadian, “The Llama 3 Herd of Models,” corr , Aug. 2024, arXiv:2407.21783 [cs]. [Online]. Available: http://arxiv.org/abs/2407.21783

Original

2024

Closest in time.

Mitre. (2024) LLM jailbreak | MITRE ATLAS™. [Online]. Available: https://atlas.mitre.org/techniques/AML.T0054

2024

Closest in time.

S. Schulhoff, J. Pinto, A. Khan, and L.-F. Bouchard, “Ignore This Title and HackAPrompt: Exposing Systemic Vulnerabilities of LLMs through a Global Scale Prompt Hacking Competition,” corr , Mar. 2024, arXiv:2311.16119 [cs]. [Online]. Available: http://arxiv.org/abs/2311.16119

Original

2024

Closest in time.

J. Rehberger. (2024) Google AI studio: LLM-powered data exfiltration hits again! quickly fixed. · embrace the red. [Online]. Available: https://embracethered.com/blog/posts/2024/google-ai-studio-data-exfiltration-now-fixed/

2024

Closest in time.

(2024) Llama 3.2: Revolutionizing edge AI and vision with open, customizable models. [Online]. Available: https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/

2024

Closest in time.

M. Abdin, J. Aneja, H. Awadalla, and A. Awadallah, “Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone,” corr , Aug. 2024, arXiv:2404.14219 [cs]. [Online]. Available: http://arxiv.org/abs/2404.14219

Original

2024

Closest in time.

G. Team, M. Riviere, S. Pathak, P. G. Sessa, and C. Hardin, “Gemma 2: Improving Open Language Models at a Practical Size,” corr , Aug. 2024, arXiv:2408.00118 [cs]. [Online]. Available: http://arxiv.org/abs/2408.00118

Original

2024

Closest in time.

A. Yang, B. Yang, B. Hui, B. Zheng, and Yu, “Qwen2 technical report,” corr , no. arXiv:2407.10671, 2024. [Online]. Available: http://arxiv.org/abs/2407.10671

Original

2024

Closest in time.

Q. Team. (2024) Qwen2.5: A party of foundation models! Section: blog. [Online]. Available: http://qwenlm.github.io/blog/qwen2.5/

2024

Closest in time.

Y. Wu, F. Roesner, T. Kohno, N. Zhang, and U. Iqbal, “SecGPT: An Execution Isolation Architecture for LLM-Based Systems,” corr , Mar. 2024, arXiv:2403.04960 [cs]. [Online]. Available: http://arxiv.org/abs/2403.04960

Original

2024

Closest in time.

(2024) Llama 3.1 | Model Cards and Prompt formats. [Online]. Available: https://www.llama.com/docs/model-cards-and-prompt-formats/llama3_1/#-tool-calling-(8b/70b/405b)-

2024

Closest in time.

OWASP. (2024) OWASP top 10 for LLM applications. OWASP. [Online]. Available: https://www.llmtop10.com

2024

Closest in time.