2020

The TrojAI Software Framework: An OpenSource tool for Embedding Trojans into Deep Learning Models

Karra, Kiran, Ashcraft, Chace, Fendley, Neil

Understand

In this paper, we introduce the TrojAI software framework, an open source set of Python tools capable of generating triggered (poisoned) datasets and associated deep learning (DL) models with trojans at scale.

We utilize the developed framework to generate a large set of trojaned MNIST classifiers, as well as demonstrate the capability to produce a trojaned reinforcement-learning model using vector observations.
Results on MNIST show that the nature of the trigger, training batch size, and dataset poisoning percentage all affect successful embedding of trojans.
We test Neural Cleanse against the trojaned MNIST models and successfully detect anomalies in the trained models approximately $18\%$ of the time.

Built on

Sepp Hochreiter and Jürgen Schmidhuber, ‘Long short-term memory’, Neural computation
1997
Earlier work this paper cites.
Diederik P Kingma and Jimmy Ba, ‘Adam: A method for stochastic optimization’, arXiv preprint arXiv:1412.6980
Original
2014
Earlier work this paper cites.
Jeffrey Pennington, Richard Socher, and Christopher D Manning, ‘Glove: Global vectors for word representation’, in Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)
2014
Earlier work this paper cites.
Openai gym, 2016
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba · 2016
Earlier work this paper cites.
Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg, ‘Badnets: Identifying vulnerabilities in the machine learning model supply chain’, arXiv preprint arXiv:1708.06733
Original
2017
Earlier work this paper cites.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger, ‘Densely connected convolutional networks’, in Proceedings of the IEEE conference on computer vision and pattern recognition
2017
Earlier work this paper cites.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, ‘Proximal policy optimization algorithms’, arXiv preprint arXiv:1707.06347
Original
2017
Earlier work this paper cites.

Similar

Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg, ‘Fine-pruning: Defending against backdooring attacks on deep neural networks’, in International Symposium on Research in Attacks, Intrusions, and Defenses
2018
Cited alongside, same era.
Alexander Turner, Dimitris Tsipras, and Aleksander Madry, ‘Clean-label backdoor attacks’, (2018)
2018
Cited alongside, same era.
Minhui Zou, Yang Shi, Chengliang Wang, Fangyu Li, WenZhan Song, and Yu Wang, ‘Potrojan: powerful neural-level trojan designs in deep learning models’, arXiv preprint arXiv:1802.03043
Original
2018
Cited alongside, same era.
Ankesh Anand, Evan Racah, Sherjil Ozair, Yoshua Bengio, Marc-Alexandre Côté, and R Devon Hjelm, ‘Unsupervised state representation learning in atari’, arXiv preprint arXiv:1906.08226
Original
2019
Cited alongside, same era.
Jiazhu Dai, Chuanshuai Chen, and Yufeng Li, ‘A backdoor attack against lstm-based text classification systems’, IEEE Access
2019
Cited alongside, same era.
Panagiota Kiourti, Kacper Wardega, Susmit Jha, and Wenchao Li, ‘Trojdrl: Trojan attacks on deep reinforcement learning agents’, arXiv preprint arXiv:1903.06638
Original
2019
Cited alongside, same era.
Yingqi Liu, Wen-Chuan Lee, Guanhong Tao, Shiqing Ma, Yousra Aafer, and Xiangyu Zhang, ‘Abs: Scanning neural networks for back-doors by artificial brain stimulation’, in Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security
2019
Cited alongside, same era.
Recurrent and multi-process PyTorch implementation of deep reinforcement Actor-Critic algorithms: A2C and PPO
Lucas Willems
Cited in the paper.

Then

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala, ‘Pytorch: An imperative style, high-performance deep learning library’, in Advances in Neural Information Processing Systems 32
2019
Later among the works it cites.
A taxonomy and terminology of adversarial machine learning, 2019
Elham Tabassi, Kevin Burns, Michael Hadjimichael, Andres Molina-Markham, and Julian Sexton · 2019
Later among the works it cites.
Te Juin Lester Tan and Reza Shokri, ‘Bypassing backdoor detection algorithms in deep learning’, arXiv preprint arXiv:1905.13409
Original
2019
Later among the works it cites.
Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y Zhao, ‘Neural cleanse: Identifying and mitigating backdoor attacks in neural networks’, in 2019 IEEE Symposium on Security and Privacy (SP)
2019
Later among the works it cites.
Zuxuan Wu, Ser-Nam Lim, Larry Davis, and Tom Goldstein, ‘Making an invisibility cloak: Real world adversarial attacks on object detectors’, arXiv preprint arXiv:1910.14667
Original
2019
Later among the works it cites.
Zhaoyuan Yang, Naresh Iyer, Johan Reimann, and Nurali Virani, ‘Design of intentional backdoors in sequential models’, arXiv preprint arXiv:1902.09972
Original
2019
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…