2020

Adversarial Machine Learning -- Industry Perspectives

Kumar, Ram Shankar Siva, Nyström, Magnus, Lambert, John et al.

Understand

Based on interviews with 28 organizations, we found that industry practitioners are not equipped with tactical and strategic tools to protect, detect and respond to attacks on their Machine Learning (ML) systems.

We leverage the insights from the interviews and we enumerate the gaps in perspective in securing machine learning systems when viewed in the context of traditional software security development.
We write this paper from the perspective of two personas: developers/ML engineers and security incident responders who are tasked with securing ML systems as they are designed, developed and deployed ML systems.
The goal of this paper is to engage researchers to revise and amend the Security Development Lifecycle for industrial-grade software in the adversarial ML era.

Built on

R. Von Solms, “Information security management: why standards are important,” Information Management & Computer Security , vol. 7, no. 1, pp. 50–58, 1999
1999
Earlier work this paper cites.
C. C. Zou, W. Gong, and D. Towsley, “Code red worm propagation modeling and analysis,” in Proceedings of the 9th ACM conference on Computer and communications security . ACM, 2002, pp. 138–147
2002
Earlier work this paper cites.
W. M. Van der Aalst and A. K. A. de Medeiros, “Process mining and security: Detecting anomalous process executions and checking process conformance,” Electronic Notes in Theoretical Computer Science , vol. 121, pp. 3–21, 2005
2005
Earlier work this paper cites.
J. Twycross, U. Aickelin, and A. Whitbrook, “Detecting anomalous process behaviour using second generation artificial immune systems,” arXiv preprint arXiv:1006.3654 , 2010
Original
2010
Earlier work this paper cites.
I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” arXiv preprint arXiv:1412.6572 , 2014
Original
2014
Earlier work this paper cites.
D. Sculley, G. Holt, D. Golovin, E. Davydov, T. Phillips, D. Ebner, V. Chaudhary, and M. Young, “Machine Learning: The High interest Credit Card of Technical Debt,” in SE4ML: Software Engineering for Machine Learning (NIPS 2014 Workshop) , 2014
2014
Earlier work this paper cites.
M. Fredrikson, S. Jha, and T. Ristenpart, “Model inversion attacks that exploit confidence information and basic countermeasures,” in Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security , 2015, pp. 1322–1333
2015
Earlier work this paper cites.
“Adversarial Machine Learning,” Jul 2016. [Online]. Available: https://ibm.co/36fhajg
2016
Earlier work this paper cites.
P. L. Microsoft, “Learning from Tay’s introduction,” Mar 2016. [Online]. Available: https://blogs.microsoft.com/blog/2016/03/25/learning-tays-introduction/
2016
Earlier work this paper cites.
D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and D. Mané, “Concrete problems in ai safety,” arXiv preprint arXiv:1606.06565 , 2016
Original
2016
Earlier work this paper cites.
F. Tramèr, F. Zhang, A. Juels, M. K. Reiter, and T. Ristenpart, “Stealing machine learning models via prediction apis,” in 25th { \{ USENIX } \} Security Symposium ( { \{ USENIX } \} Security 16) , 2016, pp. 601–618
2016
Earlier work this paper cites.
N. Papernot, P. McDaniel, A. Sinha, and M. Wellman, “Towards the science of security and privacy in machine learning,” arXiv preprint arXiv:1611.03814 , 2016
Original
2016
Earlier work this paper cites.
N. Papernot, P. McDaniel, and I. Goodfellow, “Transferability in machine learning: from phenomena to black-box attacks using adversarial samples,” arXiv preprint arXiv:1605.07277 , 2016
Original
2016
Earlier work this paper cites.
A. Athalye, L. Engstrom, A. Ilyas, and K. Kwok, “Synthesizing robust adversarial examples,” arXiv preprint arXiv:1707.07397 , 2017
Original
2017
Earlier work this paper cites.
J. Leike, M. Martic, V. Krakovna, P. A. Ortega, T. Everitt, A. Lefrancq, L. Orseau, and S. Legg, “Ai safety gridworlds,” arXiv preprint arXiv:1711.09883 , 2017
Original
2017
Earlier work this paper cites.

Similar

T. Gu, B. Dolan-Gavitt, and S. Garg, “Badnets: Identifying vulnerabilities in the machine learning model supply chain,” arXiv preprint arXiv:1708.06733 , 2017
Original
2017
Cited alongside, same era.
R. Shokri, M. Stronati, C. Song, and V. Shmatikov, “Membership inference attacks against machine learning models,” in 2017 IEEE Symposium on Security and Privacy (SP) . IEEE, 2017, pp. 3–18
2017
Cited alongside, same era.
M. Sharif, S. Bhagavatula, L. Bauer, and M. K. Reiter, “Adversarial generative nets: Neural network attacks on state-of-the-art face recognition,” arXiv preprint arXiv:1801.00349 , pp. 1556–6013, 2017
Original
2017
Cited alongside, same era.
N. Carlini and D. Wagner, “Adversarial examples are not easily detected: Bypassing ten detection methods,” in Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security . ACM, 2017, pp. 3–14
2017
Cited alongside, same era.
G. Katz, C. Barrett, D. L. Dill, K. Julian, and M. J. Kochenderfer, “Reluplex: An efficient Smt solver for verifying deep neural networks,” in International Conference on Computer Aided Verification . Springer, 2017, pp. 97–117
2017
Cited alongside, same era.
“Software 2.0,” 2017. [Online]. Available: https://medium.com/@karpathy/software-2-0-a64152b37c35
2017
Cited alongside, same era.
“2018 AI predictions 8 insights to shape business strategy,” Tech. Rep. [Online]. Available: https://www.pwc.com/us/en/advisory-services/assets/ai-predictions-2018-report.pdf
2018
Cited alongside, same era.
J. Gilmer, R. P. Adams, I. Goodfellow, D. Andersen, and G. E. Dahl, “Motivating the rules of the game for adversarial example research,” arXiv preprint arXiv:1807.06732 , 2018
Original
2018
Cited alongside, same era.
M. Jagielski, A. Oprea, B. Biggio, C. Liu, C. Nita-Rotaru, and B. Li, “Manipulating machine learning: Poisoning attacks and countermeasures for regression learning,” in 2018 IEEE Symposium on Security and Privacy (SP) . IEEE, 2018, pp. 19–35
2018
Cited alongside, same era.
G. F. Elsayed, I. Goodfellow, and J. Sohl-Dickstein, “Adversarial reprogramming of neural networks,” arXiv preprint arXiv:1806.11146 , 2018
Original
2018
Cited alongside, same era.
Q. Xiao, K. Li, D. Zhang, and W. Xu, “Security risks in deep learning implementations,” in 2018 IEEE Security and Privacy Workshops (SPW) . IEEE, 2018, pp. 123–128
2018
Cited alongside, same era.
N. Papernot, F. Faghri, N. Carlini, I. Goodfellow, R. Feinman, A. Kurakin, C. Xie, Y. Sharma, T. Brown, A. Roy, A. Matyasko, V. Behzadan, K. Hambardzumyan, Z. Zhang, Y.-L. Juang, Z. Li, R. Sheatsley, A. Garg, J. Uesato, W. Gierke, Y. Dong, D. Berthelot, P. Hendricks, J. Rauber, and R. Long, “Technical report on the cleverhans v2.1.0 adversarial examples library,” arXiv preprint arXiv:1610.00768 , 2018
Original
2018
Cited alongside, same era.

Then

N. Papernot, “A marauder’s map of security and privacy in machine learning,” arXiv preprint arXiv:1811.01134 , 2018
Original
2018
Later among the works it cites.
T.-W. Weng, H. Zhang, H. Chen, Z. Song, C.-J. Hsieh, D. Boning, I. S. Dhillon, and L. Daniel, “Towards fast computation of certified robustness for relu networks,” arXiv preprint arXiv:1804.09699 , 2018
Original
2018
Later among the works it cites.
“A Guide to Cyber Attribution,” 2018. [Online]. Available: https://bit.ly/2G50UXB
2018
Later among the works it cites.
J. Li, S. Qu, X. Li, J. Szurley, J. Z. Kolter, and F. Metze, “Adversarial Music: Real World Audio Adversary Against Wake-word Detection System,” in Advances in Neural Information Processing Systems , 2019, pp. 11 908–11 918
2019
Later among the works it cites.
“ISO/IEC JTC 1/SC 42 – Artificial Intelligence,” Jan 2019. [Online]. Available: https://www.iso.org/committee/6794475.html
2019
Later among the works it cites.
“Ethics guidelines for trustworthy ai,” Nov 2019. [Online]. Available: https://ec.europa.eu/digital-single-market/en/news/ethics-guidelines-trustworthy-ai
2019
Later among the works it cites.
D. Hendrycks and T. Dietterich, “Benchmarking neural network robustness to common corruptions and perturbations,” arXiv preprint arXiv:1903.12261 , 2019
Original
2019
Later among the works it cites.
R. S. S. Kumar, D. O. Brien, K. Albert, S. Viljöen, and J. Snover, “Failure modes in machine learning systems,” arXiv preprint arXiv:1911.11034 , 2019
Original
2019
Later among the works it cites.
[Online]. Available: https://about.fb.com/news/2019/01/designing-security-for-billions/
2019
Later among the works it cites.
[Online]. Available: https://about.fb.com/news/2019/01/designing-security-for-billions/
2019
Later among the works it cites.
N. Carlini. [Online]. Available: https://nicholas.carlini.com/writing/2019/all-adversarial-example-papers.html
2019
Later among the works it cites.
M. Melis, A. Demontis, M. Pintor, A. Sotgiu, and B. Biggio, “secml: A Python Library for Secure and Explainable Machine Learning,” arXiv preprint arXiv:1912.10013 , 2019
Original
2019
Later among the works it cites.
B. Dolhansky, R. Howes, B. Pflaum, N. Baram, and C. C. Ferrer, “The deepfake detection challenge (dfdc) preview dataset,” arXiv preprint arXiv:1910.08854 , 2019
Original
2019
Later among the works it cites.
“CVE-2020-0674.” [Online]. Available: https://kb.cert.org/vuls/id/338824/
2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…