Built on

Hellaswag: Can a machine really finish your sentence?, 2019
Original
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y · 1905
Earlier work this paper cites.
Measuring massive multitask language understanding, 2021
Original
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J · 2009
Earlier work this paper cites.
Proximal policy optimization algorithms
Original
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O · 2017
Earlier work this paper cites.
On the weaknesses of reinforcement learning for neural machine translation
Original
Choshen, L., Fox, L., Aizenbud, Z., and Abend, O · 2019
Earlier work this paper cites.
Constitutional ai: Harmlessness from ai feedback
Original
Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., et al · 2022
Earlier work this paper cites.
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al · 2022
Earlier work this paper cites.
Chatgpt: Optimizing language models for dialogue, 2022
Schulman, J., Zoph, B., Kim, C., Hilton, J., Menick, J., Weng, J., Uribe, J., Fedus, L., Metz, L., Pokorny, M., et al · 2022
Earlier work this paper cites.
Generative language models and automated influence operations: Emerging threats and potential mitigations
Original
Goldstein, J. A., Sastry, G., Musser, M., DiResta, R., Gentzel, M., and Sedova, K · 2023
Earlier work this paper cites.
Large language models can be used to effectively scale spear phishing campaigns
Original
Hazell, J · 2023
Earlier work this paper cites.
Fine-tuning aligned language models compromises safety, even when users do not intend to!
Original
Qi, X., Zeng, Y., Xie, T., Chen, P.-Y., Jia, R., Mittal, P., and Henderson, P · 2023
Earlier work this paper cites.
Stanford alpaca: An instruction-following llama model
Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B · 2023
Earlier work this paper cites.

Similar

Shadow alignment: The ease of subverting safely-aligned language models
Original
Yang, X., Wang, X., Zhang, Q., Petzold, L., Wang, W. Y., Zhao, X., and Lin, D · 2023
Cited alongside, same era.
Universal and transferable adversarial attacks on aligned language models, 2023
Original
Zou, A., Wang, Z., Carlini, N., Nasr, M., Kolter, J. Z., and Fredrikson, M · 2023
Cited alongside, same era.
Jailbreaking leading safety-aligned llms with simple adaptive attacks
Original
Andriushchenko, M., Croce, F., and Flammarion, N · 2024
Cited alongside, same era.
Many-shot jailbreaking
Anil, C., DURMUS, E., Rimsky, N., Sharma, M., Benton, J., Kundu, S., Batson, J., Tong, M., Mu, J., Ford, D. J., Mosconi, F., Agrawal, R., Schaeffer, R., Bashkansky, N., Svenningsen, S., Lambert, M., Radhakrishnan, A., Denison, C., Hubinger, E. J., Bai, Y., Bricken, T., Maxwell, T., Schiefer, N., Sully, J., Tamkin, A., Lanham, T., Nguyen, K., Korbak, T., Kaplan, J., Ganguli, D., Bowman, S. R., Perez, E., Grosse, R. B., and Duvenaud, D · 2024
Cited alongside, same era.

Then

Great, now write an article about that: The crescendo multi-turn llm jailbreak attack
Original
Russinovich, M., Salem, A., and Eldan, R · 2024
Later among the works it cites.
Xstest: A test suite for identifying exaggerated safety behaviours in large language models, 2024
Original
Röttger, P., Kirk, H. R., Vidgen, B., Attanasio, G., Bianchi, F., and Hovy, D · 2024
Later among the works it cites.
”do anything now”: Characterizing and evaluating in-the-wild jailbreak prompts on large language models, 2024
Original
Shen, X., Chen, Z., Backes, M., Shen, Y., and Zhang, Y · 2024
Later among the works it cites.
Tamper-resistant safeguards for open-weight llms, 2024
Original
Tamirisa, R., Bharathi, B., Phan, L., Zhou, A., Gatti, A., Suresh, T., Lin, M., Wang, J., Wang, R., Arel, R., Zou, A., Song, D., Li, B., Hendrycks, D., and Mazeika, M · 2024
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2025

Improving LLM Safety Alignment with Dual-Objective Optimization

Zhao, Xuandong, Cai, Will, Shi, Tianneng et al.

Understand

Existing training-time safety alignment techniques for large language models (LLMs) remain vulnerable to jailbreak attacks.

Direct preference optimization (DPO), a widely deployed alignment method, exhibits limitations in both experimental and theoretical contexts as its loss function proves suboptimal for refusal learning.
Through gradient-based analysis, we identify these shortcomings and propose an improved safety alignment that disentangles DPO objectives into two components: (1) robust refusal training, which encourages refusal even when partial unsafe generations are produced, and (2) targeted unlearning of harmful knowledge.
This approach significantly increases LLM robustness against a wide range of jailbreak attacks, including prefilling, suffix, and multi-turn attacks across both in-distribution and out-of-distribution scenarios.

Built on

Hellaswag: Can a machine really finish your sentence?, 2019
Original
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y · 1905
Earlier work this paper cites.
Measuring massive multitask language understanding, 2021
Original
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J · 2009
Earlier work this paper cites.
Proximal policy optimization algorithms
Original
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O · 2017
Earlier work this paper cites.
On the weaknesses of reinforcement learning for neural machine translation
Original
Choshen, L., Fox, L., Aizenbud, Z., and Abend, O · 2019
Earlier work this paper cites.
Constitutional ai: Harmlessness from ai feedback
Original
Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., et al · 2022
Earlier work this paper cites.
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al · 2022
Earlier work this paper cites.
Chatgpt: Optimizing language models for dialogue, 2022
Schulman, J., Zoph, B., Kim, C., Hilton, J., Menick, J., Weng, J., Uribe, J., Fedus, L., Metz, L., Pokorny, M., et al · 2022
Earlier work this paper cites.
Generative language models and automated influence operations: Emerging threats and potential mitigations
Original
Goldstein, J. A., Sastry, G., Musser, M., DiResta, R., Gentzel, M., and Sedova, K · 2023
Earlier work this paper cites.
Large language models can be used to effectively scale spear phishing campaigns
Original
Hazell, J · 2023
Earlier work this paper cites.
Fine-tuning aligned language models compromises safety, even when users do not intend to!
Original
Qi, X., Zeng, Y., Xie, T., Chen, P.-Y., Jia, R., Mittal, P., and Henderson, P · 2023
Earlier work this paper cites.
Stanford alpaca: An instruction-following llama model
Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B · 2023
Earlier work this paper cites.

Similar

Shadow alignment: The ease of subverting safely-aligned language models
Original
Yang, X., Wang, X., Zhang, Q., Petzold, L., Wang, W. Y., Zhao, X., and Lin, D · 2023
Cited alongside, same era.
Universal and transferable adversarial attacks on aligned language models, 2023
Original
Zou, A., Wang, Z., Carlini, N., Nasr, M., Kolter, J. Z., and Fredrikson, M · 2023
Cited alongside, same era.
Jailbreaking leading safety-aligned llms with simple adaptive attacks
Original
Andriushchenko, M., Croce, F., and Flammarion, N · 2024
Cited alongside, same era.
Many-shot jailbreaking
Anil, C., DURMUS, E., Rimsky, N., Sharma, M., Benton, J., Kundu, S., Batson, J., Tong, M., Mu, J., Ford, D. J., Mosconi, F., Agrawal, R., Schaeffer, R., Bashkansky, N., Svenningsen, S., Lambert, M., Radhakrishnan, A., Denison, C., Hubinger, E. J., Bai, Y., Bricken, T., Maxwell, T., Schiefer, N., Sully, J., Tamkin, A., Lanham, T., Nguyen, K., Korbak, T., Kaplan, J., Ganguli, D., Bowman, S. R., Perez, E., Grosse, R. B., and Duvenaud, D · 2024
Cited alongside, same era.

Then

Great, now write an article about that: The crescendo multi-turn llm jailbreak attack
Original
Russinovich, M., Salem, A., and Eldan, R · 2024
Later among the works it cites.
Xstest: A test suite for identifying exaggerated safety behaviours in large language models, 2024
Original
Röttger, P., Kirk, H. R., Vidgen, B., Attanasio, G., Bianchi, F., and Hovy, D · 2024
Later among the works it cites.
”do anything now”: Characterizing and evaluating in-the-wild jailbreak prompts on large language models, 2024
Original
Shen, X., Chen, Z., Backes, M., Shen, Y., and Zhang, Y · 2024
Later among the works it cites.
Tamper-resistant safeguards for open-weight llms, 2024
Original
Tamirisa, R., Bharathi, B., Phan, L., Zhou, A., Gatti, A., Suresh, T., Lin, M., Wang, J., Wang, R., Arel, R., Zou, A., Song, D., Li, B., Hendrycks, D., and Mazeika, M · 2024
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Towards analyzing and understanding the limitations of dpo: A theoretical perspective

Original

Feng, D., Qin, B., Huang, C., Zhang, Z., and Lei, W · 2024

Cited alongside, same era.

When ”competency” in reasoning opens the door to vulnerability: Jailbreaking llms via novel complex ciphers, 2024

Original

Handa, D., Zhang, Z., Saeidi, A., and Baral, C · 2024

Cited alongside, same era.

Autodan: Generating stealthy jailbreak prompts on aligned large language models, 2024

Original

Liu, X., Xu, N., Chen, M., and Xiao, C · 2024

Cited alongside, same era.

Harmbench: A standardized evaluation framework for automated red teaming and robust refusal

Original

Mazeika, M., Phan, L., Yin, X., Zou, A., Wang, Z., Mu, N., Sakhaee, E., Li, N., Basart, S., Li, B., Forsyth, D., and Hendrycks, D · 2024

Cited alongside, same era.

Safety alignment should be made more than just a few tokens deep, 2024

Original

Qi, X., Panda, A., Lyu, K., Ma, X., Roy, S., Beirami, A., Mittal, P., and Henderson, P · 2024

Cited alongside, same era.

Treebon: Enhancing inference-time alignment with speculative tree-search and best-of-n sampling

Original

Qiu, J., Lu, Y., Zeng, Y., Guo, J., Geng, J., Wang, H., Huang, K., Wu, Y., and Wang, M · 2024

Cited alongside, same era.

Direct preference optimization: Your language model is secretly a reward model

Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C · 2024

Cited alongside, same era.

Gemma 2: Improving open language models at a practical size, 2024a

Original

Team, G

Cited in the paper.

The instruction hierarchy: Training llms to prioritize privileged instructions, 2024

Original

Wallace, E., Xiao, K., Leike, R., Weng, L., Heidecke, J., and Beutel, A · 2024