2020

Trojaning Language Models for Fun and Profit

Zhang, Xinyang, Zhang, Zheng, Ji, Shouling et al.

Understand

Recent years have witnessed the emergence of a new paradigm of building natural language processing (NLP) systems: general-purpose, pre-trained language models (LMs) are composed with simple downstream models and fine-tuned for a variety of NLP tasks.

This paradigm shift significantly simplifies the system development cycles.
However, as many LMs are provided by untrusted third parties, their lack of standardization or regulation entails profound security implications, which are largely unexplored.
To bridge this gap, this work studies the security threats posed by malicious LMs to NLP systems.

Built on

C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus, “Intriguing Properties of Neural Networks,” in
2014
Earlier work this paper cites.
I. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and Harnessing Adversarial Examples,” in
2015
Earlier work this paper cites.
M. Backes, S. Bugiel, and E. Derr, “Reliable third-party library detection in android and its security applications,” in
2016
Earlier work this paper cites.
P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang, “SQuAD: 100,000+ Questions for Machine Comprehension of Text,” in
2016
Earlier work this paper cites.
N. Papernot, P. D. McDaniel, S. Jha, M. Fredrikson, Z. B. Celik, and A. Swami, “The Limitations of Deep Learning in Adversarial Settings,” in
2016
Earlier work this paper cites.
N. Papernot, P. McDaniel, X. Wu, S. Jha, and A. Swami, “Distillation as a Defense to Adversarial Perturbations Against Deep Neural Networks,” in
2016
Earlier work this paper cites.
T. Gu, B. Dolan-Gavitt, and S. Garg, “BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain,”
2017
Earlier work this paper cites.
A. Trischler, T. Wang, X. Yuan, J. Harris, A. Sordoni, P. Bachman, and K. Suleman, “NewsQA: A machine comprehension dataset,” in
2017
Earlier work this paper cites.
N. Carlini and D. A. Wagner, “Towards Evaluating the Robustness of Neural Networks,” in
2017
Earlier work this paper cites.
A. Kurakin, I. J. Goodfellow, and S. Bengio, “Adversarial Machine Learning at Scale,” in
2017
Earlier work this paper cites.
D. Meng and H. Chen, “MagNet: A Two-Pronged Defense Against Adversarial Examples,” in
2017
Earlier work this paper cites.
Y. Ji, X. Zhang, and T. Wang, “Backdoor Attacks against Learning Systems,” in
2017
Earlier work this paper cites.
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,”
2018
Earlier work this paper cites.
Y. Liu, S. Ma, Y. Aafer, W.-C. Lee, J. Zhai, W. Wang, and X. Zhang, “Trojaning attack on neural networks,” in
2018
Earlier work this paper cites.
Y. Ji, X. Zhang, S. Ji, X. Luo, and T. Wang, “Model-Reuse Attacks on Deep Learning Systems,” in
2018
Earlier work this paper cites.
A. M. Founta, C. Djouvas, D. Chatzakou, I. Leontiadis, J. Blackburn, G. Stringhini, A. Vakali, M. Sirivianos, and N. Kourtellis, “Large Scale Crowdsourcing and Characterization of Twitter Abusive Behavior,” in
2018
Earlier work this paper cites.
B. Chen, W. Carvalho, N. Baracaldo, H. Ludwig, B. Edwards, T. Lee, I. Molloy, and B. Srivastava, “Detecting Backdoor Attacks on Deep Neural Networks by Activation Clustering,” in
2018
Earlier work this paper cites.
E. Chou, F. Tramer, G. Pellegrino, and D. Boneh, “SentiNet: Detecting Physical Attacks Against Deep Learning Systems,” in
2018
Earlier work this paper cites.
B. Biggio and F. Roli, “Wild Patterns: Ten Years after The Rise of Adversarial Machine Learning,”
2018
Earlier work this paper cites.
C. Guo, M. Rana, M. Cissé, and L. van der Maaten, “Countering Adversarial Images Using Input Transformations,” in
2018
Earlier work this paper cites.
F. Tramèr, A. Kurakin, N. Papernot, I. Goodfellow, D. Boneh, and P. McDaniel, “Ensemble Adversarial Training: Attacks and Defenses,” in
2018
Earlier work this paper cites.

Similar

W. Xu, D. Evans, and Y. Qi, “Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks,” in
2018
Cited alongside, same era.
T. Gehr, M. Mirman, D. Drachsler-Cohen, P. Tsankov, S. Chaudhuri, and M. Vechev, “AI2: Safety and Robustness Certification of Neural Networks with Abstract Interpretation,” in
2018
Cited alongside, same era.
A. Athalye, N. Carlini, and D. Wagner, “Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples,” in
2018
Cited alongside, same era.
A. Shafahi, W. Ronny Huang, M. Najibi, O. Suciu, C. Studer, T. Dumitras, and T. Goldstein, “Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networks,” in
2018
Cited alongside, same era.
O. Suciu, R. Mărginean, Y. Kaya, H. Daumé, III, and T. Dumitraş, “When Does Machine Learning FAIL? Generalized Transferability for Evasion and Poisoning Attacks,” in
2018
Cited alongside, same era.
B. Tran, J. Li, and A. Madry, “Spectral Signatures in Backdoor Attacks,” in
2018
Cited alongside, same era.
J. Ebrahimi, A. Rao, D. Lowd, and D. Dou, “HotFlip: White-box adversarial examples for text classification,” in
2018
Cited alongside, same era.
M. Alzantot, Y. Sharma, A. Elgohary, B.-J. Ho, M. Srivastava, and K.-W. Chang, “Generating natural language adversarial examples,” in
2018
Cited alongside, same era.
J. Ebrahimi, D. Lowd, and D. Dou, “On adversarial examples for character-level neural machine translation,” in
2018
Cited alongside, same era.
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Language Models Are Unsupervised Multitask Learners,” OpenAI Technical Report, 2019
2019
Cited alongside, same era.
Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. Salakhutdinov, and Q. V. Le, “XLNet: Generalized Autoregressive Pretraining for Language Understanding,” in
2019
Cited alongside, same era.
Y. Yao, H. Li, H. Zheng, and B. Y. Zhao, “Latent Backdoor Attacks on Deep Neural Networks,” in
2019
Cited alongside, same era.

Then

Y. Gao, C. Xu, D. Wang, S. Chen, D. C. Ranasinghe, and S. Nepal, “Strip: A defence against trojan attacks on deep neural networks,” in
2019
Later among the works it cites.
B. Wang, Y. Yao, S. Shan, H. Li, B. Viswanath, H. Zheng, and B. Y. Zhao, “Neural Cleanse: Identifying and Mitigating Backdoor Attacks in Neural Networks,” in
2019
Later among the works it cites.
H. Chen, C. Fu, J. Zhao, and F. Koushanfar, “DeepInspect: A Black-box Trojan Detection and Mitigation Framework for Deep Neural Networks,” in
2019
Later among the works it cites.
Y. Liu, W.-C. Lee, G. Tao, S. Ma, Y. Aafer, and X. Zhang, “ABS: Scanning Neural Networks for Back-Doors by Artificial Brain Stimulation,” in
2019
Later among the works it cites.
S. Ma, Y. Liu, G. Tao, W.-C. Lee, and X. Zhang, “NIC: Detecting Adversarial Samples with Neural Network Invariant Checking,” in
2019
Later among the works it cites.
X. Ling, S. Ji, J. Zou, J. Wang, C. Wu, B. Li, and T. Wang, “DEEPSEC: A Uniform Platform for Security Analysis of Deep Learning Model,” in
2019
Later among the works it cites.
J. Li, S. Ji, T. Du, B. Li, and T. Wang, “TextBugger: Generating Adversarial Text Against Real-world Applications,” in
2019
Later among the works it cites.
S. Ren, Y. Deng, K. He, and W. Che, “Generating natural language adversarial examples through probability weighted word saliency,” in
2019
Later among the works it cites.
R. Jia, A. Raghunathan, K. Göksel, and P. Liang, “Certified robustness to adversarial word substitutions,” in
2019
Later among the works it cites.
H. Zhong, C. Xiao, C. Tu, T. Zhang, Z. Liu, and M. Sun, “How does NLP benefit legal system: A summary of legal artificial intelligence,” in
2020
Closest in time.
K. Kurita, P. Michel, and G. Neubig, “Weight Poisoning Attacks on Pre-trained Models,” in
2020
Closest in time.
S. Dathathri, A. Madotto, J. Lan, J. Hung, E. Frank, P. Molino, J. Yosinski, and R. Liu, “Plug and play language models: A simple approach to controlled text generation,” in
2020
Closest in time.
A. Holtzman, J. Buys, L. Du, M. Forbes, and Y. Choi, “The curious case of neural text degeneration,” in
2020
Closest in time.
C. Yun, S. Bhojanapalli, A. Singh Rawat, S. J. Reddi, and S. Kumar, “Are Transformers universal approximators of sequence-to-sequence functions?” in
2020
Closest in time.
B. Doan, E. Abbasnejad, and D. Ranasinghe, “Februus: Input Purification Defense Against Trojan Attacks on Deep Neural Network Systems,” in
2020
Closest in time.
M. Cheng, J. Yi, P.-Y. Chen, H. Zhang, and C.-J. Hsieh, “Seq2sick: Evaluating the robustness of sequence-to-sequence models with adversarial examples.” in
2020
Closest in time.
J. Li, T. Du, S. Ji, R. Zhang, Q. Lu, M. Yang, and T. Wang, “Textshield: Robust text classification based on multimodal embedding and neural machine translation,” in
2020
Closest in time.
R. Schuster, T. Schuster, Y. Meri, and V. Shmatikov, “Humpty Dumpty: Controlling Word Meanings via Corpus Poisoning,” in
2020
Closest in time.
X. Chen, A. Salem, M. Backes, S. Ma, and Y. Zhang, “Badnl: Backdoor attacks against nlp models,”
Original
2020
Closest in time.
R. Pang, H. Shen, X. Zhang, S. Ji, Y. Vorobeychik, X. Luo, A. Liu, and T. Wang, “A Tale of Evil Twins: Adversarial Inputs versus Poisoned Models,” in
2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…