Fetching the paper…

Reading the bibliography…

2020

Towards Practical Lottery Ticket Hypothesis for Adversarial Training

Li, Bai, Wang, Shiqi, Jia, Yunhan et al.

Original
PDF
Post on X

Understand

Recent research has proposed the lottery ticket hypothesis, suggesting that for a deep neural network, there exist trainable sub-networks performing equally or better than the original model with commensurate training steps.

While this discovery is insightful, finding proper sub-networks requires iterative training and pruning.
The high cost incurred limits the applications of the lottery ticket hypothesis.
We show there exists a subset of the aforementioned sub-networks that converge significantly faster during the training process and thus can mitigate the cost issue.

Towards Practical Lottery Ticket Hypothesis for Adversarial Training · Around

Built on

Optimal brain damage
Yann LeCun, John S Denker, and Sara A Solla · 1990
Earlier work this paper cites.
Intriguing properties of neural networks
Original
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus · 2013
Earlier work this paper cites.
Do deep nets really need to be deep?
Jimmy Ba and Rich Caruana · 2014
Earlier work this paper cites.
Very deep convolutional networks for large-scale image recognition
Original
Karen Simonyan and Andrew Zisserman · 2014
Earlier work this paper cites.
Explaining and harnessing adversarial examples
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy · 2015
Earlier work this paper cites.
Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding
Original
Song Han, Huizi Mao, and William J Dally · 2015
Earlier work this paper cites.
Learning both weights and connections for efficient neural network
Song Han, Jeff Pool, John Tran, and William Dally · 2015
Earlier work this paper cites.
Distilling the knowledge in a neural network
Original
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean · 2015
Earlier work this paper cites.
Batch normalization: Accelerating deep network training by reducing internal covariate shift
Sergey Ioffe and Christian Szegedy · 2015
Earlier work this paper cites.
Dynamic network surgery for efficient dnns
Yiwen Guo, Anbang Yao, and Yurong Chen · 2016
Earlier work this paper cites.
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2016
Earlier work this paper cites.

Similar

Adversarial machine learning at scale
Original
Alexey Kurakin, Ian Goodfellow, and Samy Bengio · 2016
Cited alongside, same era.
Pruning filters for efficient convnets
Original
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf · 2016
Cited alongside, same era.
Delving into transferable adversarial examples and black-box attacks
Original
Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song · 2016
Cited alongside, same era.
Transferability in machine learning: from phenomena to black-box attacks using adversarial samples
Original
Nicolas Papernot, Patrick McDaniel, and Ian Goodfellow · 2016
Cited alongside, same era.
Wide residual networks

Then

Snip: Single-shot network pruning based on connection sensitivity
Original
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr · 2018
Later among the works it cites.
Rethinking the value of network pruning
Original
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell · 2018
Later among the works it cites.
To compress or not to compress: Understanding the interactions between adversarial attacks and neural network compression
Original
Yiren Zhao, Ilia Shumailov, Robert Mullins, and Ross Anderson · 2018
Later among the works it cites.
The difficulty of training sparse neural networks
Original
Utku Evci, Fabian Pedregosa, Aidan Gomez, and Erich Elsen · 2019
Later among the works it cites.
The lottery ticket hypothesis: Finding sparse, trainable neural networks

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Sergey Zagoruyko and Nikos Komodakis · 2016

Cited alongside, same era.

Less is more: Towards compact cnns

Hao Zhou, Jose M Alvarez, and Fatih Porikli · 2016

Cited alongside, same era.

Towards evaluating the robustness of neural networks

Nicholas Carlini and David Wagner · 2017

Cited alongside, same era.

Channel pruning for accelerating very deep neural networks

Yihui He, Xiangyu Zhang, and Jian Sun · 2017

Cited alongside, same era.

Towards deep learning models resistant to adversarial attacks

Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu · 2017

Cited alongside, same era.

Stochastic activation pruning for robust adversarial defense

Guneet S Dhillon, Kamyar Azizzadenesheli, Zachary C Lipton, Jeremy Bernstein, Jean Kossaifi, Aran Khanna, and Anima Anandkumar · 2018

Cited alongside, same era.

Sparse dnns with improved adversarial robustness

Yiwen Guo, Chao Zhang, Changshui Zhang, and Yurong Chen · 2018

Cited alongside, same era.

Jonathan Frankle and Michael Carbin · 2019

Later among the works it cites.

The lottery ticket hypothesis at scale

Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin · 2019

Later among the works it cites.

Towards compact and robust deep neural networks

Vikash Sehwag, Shiqi Wang, Prateek Mittal, and Suman Jana · 2019

Later among the works it cites.

Adversarial training for free!

Ali Shafahi, Mahyar Najibi, Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry S Davis, Gavin Taylor, and Tom Goldstein · 2019

Later among the works it cites.

Towards robust compressed convolutional neural networks

Arie Wahyu Wijayanto, Jun Jin Choong, Kaushalya Madhawa, and Tsuyoshi Murata · 2019

Later among the works it cites.

Second rethinking of network pruning in the adversarial setting

Shaokai Ye, Kaidi Xu, Sijia Liu, Hao Cheng, Jan-Henrik Lambrechts, Huan Zhang, Aojun Zhou, Kaisheng Ma, Yanzhi Wang, and Xue Lin · 2019

Later among the works it cites.