2023

Beyond the Safeguards: Exploring the Security Risks of ChatGPT

Derner, Erik, Batistič, Kristina

Understand

The increasing popularity of large language models (LLMs) such as ChatGPT has led to growing concerns about their safety, security risks, and ethical implications.

This paper aims to provide an overview of the different types of security risks associated with ChatGPT, including malicious text and code generation, private data disclosure, fraudulent services, information gathering, and producing unethical content.
We present an empirical study examining the effectiveness of ChatGPT's content filters and explore potential ways to bypass these safeguards, demonstrating the ethical implications and security risks that persist in LLMs even when protections are in place.
Based on a qualitative analysis of the security implications, we discuss potential strategies to mitigate these risks and inform researchers, policymakers, and industry professionals about the complex security challenges posed by LLMs like ChatGPT.

Built on

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention Is All You Need,” Advances in Neural Information Processing Systems , vol. 30, 2017
2017
Earlier work this paper cites.
P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei, “Deep Reinforcement Learning from Human Preferences,” Advances in Neural Information Processing Systems , vol. 30, 2017
2017
Earlier work this paper cites.
S. Hisamoto, M. Post, and K. Duh, “Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data in Your Machine Translation System?” Transactions of the Association for Computational Linguistics , vol. 8, pp. 49–63, 2020
2020
Earlier work this paper cites.
X. Pan, M. Zhang, S. Ji, and M. Yang, “Privacy Risks of General-Purpose Language Models,” in 2020 IEEE Symposium on Security and Privacy (SP) . IEEE, 2020, pp. 1314–1331
2020
Earlier work this paper cites.
J. Dhamala, T. Sun, V. Kumar, S. Krishna, Y. Pruksachatkun, K.-W. Chang, and R. Gupta, “BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation,” in Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency , 2021, pp. 862–872
2021
Earlier work this paper cites.
P. P. Liang, C. Wu, L.-P. Morency, and R. Salakhutdinov, “Towards Understanding and Mitigating Social Biases in Language Models,” in International Conference on Machine Learning . PMLR, 2021, pp. 6565–6576
2021
Earlier work this paper cites.
Y. Chen, C. Mahoney, I. Grasso, E. Wali, A. Matthews, T. Middleton, M. Njie, and J. Matthews, “Gender Bias and Under-Representation in Natural Language Processing Across Human Languages,” in Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society , 2021, pp. 24–34
2021
Earlier work this paper cites.
A. Tamkin, M. Brundage, J. Clark, and D. Ganguli, “Understanding the Capabilities, Limitations, and Societal Impact of Large Language Models,” arXiv preprint arXiv:2102.02503 , 2021
Original
2021
Earlier work this paper cites.
R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, et al. , “On the Opportunities and Risks of Foundation Models,” arXiv preprint arXiv:2108.07258 , 2021
Original
2021
Earlier work this paper cites.
W. I. Cho, J. Kim, J. Yang, and N. S. Kim, “Towards Cross-Lingual Generalization of Translation Gender Bias,” in Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency , 2021, pp. 449–457
2021
Earlier work this paper cites.
H. R. Kirk, Y. Jun, F. Volpin, H. Iqbal, E. Benussi, F. Dreyer, A. Shtedritski, and Y. Asano, “Bias Out-of-the-Box: An Empirical Analysis of Intersectional Occupational Biases in Popular Generative Language Models,” Advances in Neural Information Processing Systems , vol. 34, pp. 2611–2624, 2021
2021
Earlier work this paper cites.
A. Xu, E. Pathak, E. Wallace, S. Gururangan, M. Sap, and D. Klein, “Detoxifying Language Models Risks Marginalizing Minority Voices,” in Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , 2021, pp. 2390–2397
2021
Earlier work this paper cites.
H. Ngo, C. Raterink, J. G. Araújo, I. Zhang, C. Chen, A. Morisot, and N. Frosst, “Mitigating Harm in Language Models with Conditional-Likelihood Filtration,” arXiv preprint arXiv:2108.07790 , 2021
Original
2021
Earlier work this paper cites.
N. Carlini, F. Tramèr, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. Brown, D. Song, Ú. Erlingsson, A. Oprea, and C. Raffel, “Extracting Training Data from Large Language Models,” in 30th USENIX Security Symposium (USENIX Security 21) . USENIX Association, Aug 2021, pp. 2633–2650
2021
Earlier work this paper cites.

Similar

M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. , “Evaluating Large Language Models Trained on Code,” arXiv preprint arXiv:2107.03374 , 2021
Original
2021
Cited alongside, same era.
H. S. Heidenreich and J. R. Williams, “The Earth is Flat and the Sun is not a Star: The Susceptibility of GPT-2 to Universal Adversarial Triggers,” in Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society , 2021, pp. 566–573
2021
Cited alongside, same era.
P. Ranade, A. Piplai, S. Mittal, A. Joshi, and T. Finin, “Generating Fake Cyber Threat Intelligence Using Transformer-Based Models,” in 2021 International Joint Conference on Neural Networks (IJCNN) . IEEE, 2021, pp. 1–9
2021
Cited alongside, same era.
T. Schick, S. Udupa, and H. Schütze, “Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP,” Transactions of the Association for Computational Linguistics , vol. 9, pp. 1408–1424, 2021
2021
Cited alongside, same era.
OpenAI. (2022, Nov) Introducing ChatGPT. [Online]. Available: https://openai.com/blog/chatgpt
2022
Cited alongside, same era.
L. Weidinger, J. Uesato, M. Rauh, C. Griffin, P.-S. Huang, J. Mellor, A. Glaese, M. Cheng, B. Balle, A. Kasirzadeh, et al. , “Taxonomy of Risks Posed by Language Models,” in 2022 ACM Conference on Fairness, Accountability, and Transparency , 2022, pp. 214–229
2022
Cited alongside, same era.
S. Lin, J. Hilton, and O. Evans, “TruthfulQA: Measuring How Models Mimic Human Falsehoods,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , 2022, pp. 3214–3252
2022
Cited alongside, same era.
W. Cai, R. Encarnacion, B. Chern, S. Corbett-Davies, M. Bogen, S. Bergman, and S. Goel, “Adaptive Sampling Strategies to Construct Equitable Training Datasets,” in 2022 ACM Conference on Fairness, Accountability, and Transparency , 2022, pp. 1467–1478
2022
Cited alongside, same era.
H. Hu, Z. Salcic, L. Sun, G. Dobbie, P. S. Yu, and X. Zhang, “Membership Inference Attacks on Machine Learning: A Survey,” ACM Computing Surveys (CSUR) , vol. 54, no. 11s, pp. 1–37, 2022
2022
Cited alongside, same era.
H. Brown, K. Lee, F. Mireshghallah, R. Shokri, and F. Tramèr, “What Does it Mean for a Language Model to Preserve Privacy?” in 2022 ACM Conference on Fairness, Accountability, and Transparency , 2022, pp. 2280–2292
2022
Cited alongside, same era.
R. Karanjai, “Targeted Phishing Campaigns using Large Scale Language Models,” arXiv preprint arXiv:2301.00665 , 2022
Original
2022
Cited alongside, same era.
E. Perez, S. Huang, F. Song, T. Cai, R. Ring, J. Aslanides, A. Glaese, N. McAleese, and G. Irving, “Red Teaming Language Models with Language Models,” arXiv preprint arXiv:2202.03286 , 2022
Original
2022
Cited alongside, same era.

Then

J. Deng, H. Sun, Z. Zhang, J. Cheng, and M. Huang, “Recent Advances Towards Safe, Responsible, and Moral Dialogue Systems: A Survey,” arXiv preprint arXiv:2302.09270 , 2023
Original
2023
Closest in time.
M. Brundage, K. Mayer, T. Eloundou, S. Agarwal, S. Adler, G. Krueger, J. Leike, and P. Mishkin. (2023, Mar) Lessons Learned on Language Model Safety and Misuse. [Online]. Available: {https://openai.com/research/language-model-safety-and-misuse}
2023
Closest in time.
P. Hacker, A. Engel, and M. Mauer, “Regulating ChatGPT and Other Large Generative AI Models,” arXiv preprint arXiv:2302.02337 , 2023
Original
2023
Closest in time.
E. A. van Dis, J. Bollen, W. Zuidema, R. van Rooij, and C. L. Bockting, “ChatGPT: Five Priorities for Research,” Nature , vol. 614, no. 7947, pp. 224–226, 2023
2023
Closest in time.
T. Y. Zhuo, Y. Huang, C. Chen, and Z. Xing, “Exploring AI Ethics of ChatGPT: A Diagnostic Analysis,” arXiv preprint arXiv:2301.12867 , 2023
Original
2023
Closest in time.
J. Rudolph, S. Tan, and S. Tan, “ChatGPT: Bullshit Spewer or the End of Traditional Assessments in Higher Education?” Journal of Applied Learning and Teaching , vol. 6, no. 1, 2023
2023
Closest in time.
H. Hajipour, T. Holz, L. Schönherr, and M. Fritz, “Systematically Finding Security Vulnerabilities in Black-Box Code Generation Models,” arXiv preprint arXiv:2302.04012 , 2023
Original
2023
Closest in time.
D. Kang, X. Li, I. Stoica, C. Guestrin, M. Zaharia, and T. Hashimoto, “Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks,” arXiv preprint arXiv:2302.05733 , 2023
Original
2023
Closest in time.
G. Sebastian, “Do ChatGPT and Other AI Chatbots Pose a Cybersecurity Risk? – An Exploratory Study,” SSRN Preprint , p. 1023, Feb 2023
2023
Closest in time.
J. Shi, Y. Liu, P. Zhou, and L. Sun, “BadGPT: Exploring Security Vulnerabilities of ChatGPT via Backdoor Attacks to InstructGPT,” Feb 2023, poster presented at the 30th Network and Distributed System Security (NDSS) Symposium
2023
Closest in time.
OpenAI, “GPT-4 Technical Report,” OpenAI, Tech. Rep., Mar 2023
2023
Closest in time.
G. K.-M. Liu, “Perspectives on the Social Impacts of Reinforcement Learning with Human Feedback,” arXiv preprint arXiv:2303.02891 , 2023
Original
2023
Closest in time.
H. Devinney, J. Björklund, and H. Björklund, “Theories of “Gender” in NLP Bias Research,” in 2022 ACM Conference on Fairness, Accountability, and Transparency , 2022, pp. 2083–2102
2083
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…