2024

Exposing Privacy Gaps: Membership Inference Attack on Preference Data for LLM Alignment

Feng, Qizhang, Kasa, Siva Rajesh, Kasa, Santhosh Kumar et al.

Understand

Large Language Models (LLMs) have seen widespread adoption due to their remarkable natural language capabilities.

However, when deploying them in real-world settings, it is important to align LLMs to generate texts according to acceptable human standards.
Methods such as Proximal Policy Optimization (PPO) and Direct Preference Optimization (DPO) have enabled significant progress in refining LLMs using human preference data.
However, the privacy concerns inherent in utilizing such preference data have yet to be adequately studied.

Built on

Bertscore: Evaluating text generation with bert
Original
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., and Artzi, Y. (2019) · 1904
Earlier work this paper cites.
Studies in language behavior: A program of research
Johnson, W. (1944) · 1944
Earlier work this paper cites.
Bleu: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. (2002) · 2002
Earlier work this paper cites.
Rouge: A package for automatic evaluation of summaries
Lin, C.-Y. (2004) · 2004
Earlier work this paper cites.
Meteor: An automatic metric for mt evaluation with improved correlation with human judgments
Banerjee, S. and Lavie, A. (2005) · 2005
Earlier work this paper cites.
Extracting training data from large language models. corr abs/2012.07805 (2020)
Original
Carlini, N., Tramèr, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T. B., Song, D., Erlingsson, Ú., et al. (2020) · 2012
Earlier work this paper cites.
Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding
Original
Han, S., Mao, H., and Dally, W. J. (2015) · 2015
Earlier work this paper cites.
A diversity-promoting objective function for neural conversation models
Original
Li, J., Galley, M., Brockett, C., Gao, J., and Dolan, B. (2015) · 2015
Earlier work this paper cites.
Deep learning with differential privacy
Abadi, M., Chu, A., Goodfellow, I., McMahan, H. B., Mironov, I., Talwar, K., and Zhang, L. (2016) · 2016
Earlier work this paper cites.
Deep reinforcement learning from human preferences
Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. (2017) · 2017
Earlier work this paper cites.
Proximal policy optimization algorithms
Original
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017) · 2017
Earlier work this paper cites.
Membership inference attacks against machine learning models
Shokri, R., Stronati, M., Song, C., and Shmatikov, V. (2017) · 2017
Earlier work this paper cites.
Comprehensive privacy analysis of deep learning
Nasr, M., Shokri, R., and Houmansadr, A. (2018) · 2018
Earlier work this paper cites.
Privacy risk in machine learning: Analyzing the connection to overfitting
Yeom, S. et al. (2018) · 2018
Earlier work this paper cites.
The secret sharer: Evaluating and testing unintended memorization in neural networks
Carlini, N., Liu, C., Erlingsson, Ú., Kos, J., and Song, D. (2019) · 2019
Earlier work this paper cites.

Similar

Language models are unsupervised multitask learners
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. (2019) · 2019
Cited alongside, same era.
White-box vs black-box: Bayes optimal strategies for membership inference
Sablayrolles, A., Douze, M., Schmid, C., Ollivier, Y., and Jégou, H. (2019) · 2019
Cited alongside, same era.
Learning to summarize with human feedback
Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. F. (2020) · 2020
Cited alongside, same era.
Extracting training data from large language models
Carlini, N. et al. (2021) · 2021
Cited alongside, same era.
Unsolved problems in ml safety
Original
Hendrycks, D., Carlini, N., Schulman, J., and Steinhardt, J. (2021) · 2021
Cited alongside, same era.
Memorisation versus generalisation in pre-trained language models
Original
Tänzer, M., Ruder, S., and Rei, M. (2021) · 2021
Cited alongside, same era.
Training a helpful and harmless assistant with reinforcement learning from human feedback
Original
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. (2022) · 2022
Cited alongside, same era.
Relaxloss: Defending membership inference attacks without losing utility
Chen, D., Yu, N., and Fritz, M. (2022) · 2022
Cited alongside, same era.
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · 2022
Cited alongside, same era.
On the amplification of security and privacy risks by post-hoc explanations in machine learning models
Original
Quan, P., Chakraborty, S., Jeyakumar, J. V., and Srivastava, M. (2022) · 2022
Cited alongside, same era.
Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization
Original
Ramamurthy, R., Ammanabrolu, P., Brantley, K., Hessel, J., Sifa, R., Bauckhage, C., Hajishirzi, H., and Choi, Y. (2022) · 2022
Cited alongside, same era.
Fundamental limits of membership inference attacks on machine learning models
Original
Aubinais, E., Gassiat, E., and Piantanida, P. (2023) · 2023
Cited alongside, same era.

Then

Policy optimization in rlhf: The impact of out-of-preference data
Original
Li, Z., Xu, T., and Yu, Y. (2023) · 2023
Later among the works it cites.
Detecting pretraining data from large language models
Original
Shi, W., Ajith, A., Xia, M., Huang, Y., Liu, D., Blevins, T., Chen, D., and Zettlemoyer, L. (2023) · 2023
Later among the works it cites.
Llama: Open and efficient foundation language models
Original
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · 2023
Later among the works it cites.
Trl: Transformer reinforcement learning
von Werra, L., Belkada, Y., Tunstall, L., Beeching, E., Thrush, T., Lambert, N., Huang, S., Rasul, K., and Gallouédec, Q. (2023) · 2023
Later among the works it cites.
A survey of large language models
Original
Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. (2023) · 2023
Later among the works it cites.
Do membership inference attacks work on large language models?
Original
Duan, M., Suri, A., Mireshghallah, N., Min, S., Shi, W., Zettlemoyer, L., Tsvetkov, Y., Choi, Y., Evans, D., and Hajishirzi, H. (2024) · 2024
Closest in time.
Unpacking dpo and ppo: Disentangling best practices for learning from preference feedback
Ivison, H., Wang, Y., Liu, J., Wu, Z., Pyatkin, V., Lambert, N., Smith, N. A., Choi, Y., and Hajishirzi, H. (2024) · 2024
Closest in time.
Llm dataset inference: Did you train on my dataset?
Original
Maini, P., Jia, H., Papernot, N., and Dziedzic, A. (2024) · 2024
Closest in time.
Direct preference optimization: Your language model is secretly a reward model
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. (2024) · 2024
Closest in time.
Detecting pretraining data from large language models
Shi, W., Ajith, A., Xia, M., Huang, Y., Liu, D., Blevins, T., Chen, D., and Zettlemoyer, L. (2024) · 2024
Closest in time.
Memorization in deep learning: A survey
Original
Wei, J., Zhang, Y., Zhang, L. Y., Ding, M., Chen, C., Ong, K.-L., Zhang, J., and Xiang, Y. (2024) · 2024
Closest in time.
Is dpo superior to ppo for llm alignment? a comprehensive study
Original
Xu, S., Fu, W., Gao, J., Ye, W., Liu, W., Mei, Z., Wang, G., Yu, C., and Wu, Y. (2024) · 2024
Closest in time.
Harnessing the power of llms in practice: A survey on chatgpt and beyond
Yang, J., Jin, H., Tang, R., Han, X., Feng, Q., Jiang, H., Zhong, S., Yin, B., and Hu, X. (2024) · 2024
Closest in time.
Min-k%++: Improved baseline for detecting pre-training data from large language models
Original
Zhang, J., Sun, J., Yeats, E., Ouyang, Y., Kuo, M., Zhang, J., Yang, H., and Li, H. (2024) · 2024
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…