Fetching the paper…

Reading the bibliography…

2023

Enhancing Adversarial Training via Reweighting Optimization Trajectory

Huang, Tianjin, Liu, Shiwei, Chen, Tianlong et al.

Original
PDF
Post on X

Understand

Despite the fact that adversarial training has become the de facto method for improving the robustness of deep neural networks, it is well-known that vanilla adversarial training suffers from daunting robust overfitting, resulting in unsatisfactory robust generalization.

A number of approaches have been proposed to address these drawbacks such as extra regularization, adversarial weights perturbation, and training with more data over the last few years.
However, the robust generalization improvement is yet far from satisfactory.
In this paper, we approach this challenge with a brand new perspective -- refining historical optimization trajectories.

Enhancing Adversarial Training via Reweighting Optimization Trajectory · Around

Built on

Qin, C., Martens, J., Gowal, S., Krishnan, D., Krishnamurthy, Dvijotham, Fawzi, A., De, S., Stanforth, R., Kohli, P.: Adversarial Robustness through Local Linearization (jul 2019), http://arxiv.org/abs/1907.02610
Original
1907
Earlier work this paper cites.
Elisseeff, A., Evgeniou, T., Pontil, M., Kaelbing, L.P.: Stability of randomized learning algorithms. Journal of Machine Learning Research 6
2005
Earlier work this paper cites.
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition. pp. 248–255. Ieee (2009)
2009
Earlier work this paper cites.
Krizhevsky, A., Nair, V., Hinton, G.: Cifar-10 (canadian institute for advanced research). URL http://www. cs. toronto. edu/kriz/cifar. html 5
2010
Earlier work this paper cites.
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., Ng, A.Y.: Reading digits in natural images with unsupervised feature learning (2011)
2011
Earlier work this paper cites.
Hinton, G., Deng, L., Yu, D., Dahl, G.E., Mohamed, A.r., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T.N., Kingsbury, B.: Deep Neural Networks for Acoustic Modeling in Speech Recognition. Ieee Signal Processing Magazine (2012). https://doi.org/10.1109/MSP.2012.2205597
2012
Earlier work this paper cites.
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., Fergus, R.: Intriguing properties of neural networks (dec 2013), http://arxiv.org/abs/1312.6199
Original
2013
Earlier work this paper cites.
Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (2014). https://doi.org/10.1109/CVPR.2014.81
2014
Earlier work this paper cites.
Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and Harnessing Adversarial Examples (dec 2014), http://arxiv.org/abs/1412.6572
Original
2014
Earlier work this paper cites.
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
Original
2014
Earlier work this paper cites.
Hardt, M., Recht, B., Singer, Y.: Train faster, generalize better: Stability of stochastic gradient descent. In: International conference on machine learning. pp. 1225–1234. PMLR (2016)
2016
Earlier work this paper cites.
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)
2016
Earlier work this paper cites.
Moosavi-Dezfooli, S.M., Fawzi, A., Frossard, P.: DeepFool: A Simple and Accurate Method to Fool Deep Neural Networks. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. vol. 2016-Decem, pp. 2574–2582. IEEE Computer Society (dec 2016). https://doi.org/10.1109/CVPR.2016.282
2016
Earlier work this paper cites.
Rhu, M., Gimelshein, N., Clemons, J., Zulfiqar, A., Keckler, S.W.: vdnn: Virtualized deep neural networks for scalable, memory-efficient neural network design. In: 2016 49th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO). pp. 1–13. IEEE (2016)
2016
Earlier work this paper cites.
Zagoruyko, S., Komodakis, N.: Wide residual networks. arXiv preprint arXiv:1605.07146 (2016)
Original
2016
Earlier work this paper cites.
Carlini, N., Wagner, D.: Towards evaluating the robustness of neural networks. In: 2017 IEEE Symposium on Security and Privacy (SP). pp. 39–57. IEEE (2017)
2017
Earlier work this paper cites.
Dziugaite, G.K., Roy, D.M.: Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. arXiv preprint arXiv:1703.11008 (2017)
Original
2017
Earlier work this paper cites.
Madry, A., Makelov, A., Schmidt, L., Tsipras, D., Vladu, A.: Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083 (2017)
Original
2017
Earlier work this paper cites.
Madry, A., Makelov, A., Schmidt, L., Tsipras, D., Vladu, A.: Towards Deep Learning Models Resistant to Adversarial Attacks (jun 2017), http://arxiv.org/abs/1706.06083
Original
2017
Earlier work this paper cites.

Similar

Zhang, C., Liao, Q., Rakhlin, A., Sridharan, K., Miranda, B., Golowich, N., Poggio, T.: Musings on deep learning: Properties of sgd. Tech. rep., Center for Brains, Minds and Machines (CBMM) (2017)
2017
Cited alongside, same era.
Athalye, A., Carlini, N., Wagner, D.: Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. arXiv preprint arXiv:1802.00420 (2018)
Original
2018
Cited alongside, same era.
Engstrom, L., Ilyas, A., Athalye, A.: Evaluating and understanding the robustness of adversarial logit pairing. arXiv preprint arXiv:1807.10272 (2018)
Original
2018
Cited alongside, same era.
Moosavi-Dezfooli, S.M., Fawzi, A., Uesato, J., Frossard, P.: Robustness via curvature regularization, and vice versa (nov 2018), http://arxiv.org/abs/1811.09716
Original
2018
Cited alongside, same era.

Then

Croce, F., Hein, M.: Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In: International conference on machine learning. pp. 2206–2216. PMLR (2020)
2020
Later among the works it cites.
Foret, P., Kleiner, A., Mobahi, H., Neyshabur, B.: Sharpness-aware minimization for efficiently improving generalization. arXiv preprint arXiv:2010.01412 (2020)
Original
2020
Later among the works it cites.
Huang, T., Menkovski, V., Pei, Y., Pechenizkiy, M.: Bridging the performance gap between fgsm and pgd adversarial training. arXiv preprint arXiv:2011.05157 (2020)
Original
2020
Later among the works it cites.
Rice, L., Wong, E., Kolter, Z.: Overfitting in adversarially robust deep learning. In: International Conference on Machine Learning. pp. 8093–8104. PMLR (2020)
2020
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Ross, A., Doshi-Velez, F.: Improving the adversarial robustness and interpretability of deep neural networks by regularizing their input gradients. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 32 (2018)

2018

Cited alongside, same era.

Schmidt, L., Santurkar, S., Tsipras, D., Talwar, K., Madry, A.: Adversarially robust generalization requires more data. In: Advances in Neural Information Processing Systems. pp. 5014–5026 (2018)

2018

Cited alongside, same era.

Uesato, J., O’donoghue, B., Kohli, P., Oord, A.: Adversarial risk and the dangers of evaluating against weak attacks. In: International Conference on Machine Learning. pp. 5025–5034. PMLR (2018)

2018

Cited alongside, same era.

Zhou, Y., Liang, Y., Zhang, H.: Generalization error bounds with probabilistic guarantee for sgd in nonconvex optimization. arXiv preprint arXiv:1802.06903 (2018)

2018

Cited alongside, same era.

Alayrac, J.B., Uesato, J., Huang, P.S., Fawzi, A., Stanforth, R., Kohli, P.: Are labels required for improving adversarial robustness? Advances in Neural Information Processing Systems 32

2019

Cited alongside, same era.

Balaji, Y., Goldstein, T., Hoffman, J.: Instance adaptive adversarial training: Improved accuracy tradeoffs in neural nets. arXiv preprint arXiv:1910.08051 (2019)

2019

Cited alongside, same era.

Carmon, Y., Raghunathan, A., Schmidt, L., Duchi, J.C., Liang, P.S.: Unlabeled data improves adversarial robustness. Advances in Neural Information Processing Systems 32

2019

Cited alongside, same era.

Cohen, J.M., Rosenfeld, E., Kolter, J.Z.: Certified adversarial robustness via randomized smoothing. arXiv preprint arXiv:1902.02918 (2019)

2019

Cited alongside, same era.

Wang, Y., Zou, D., Yi, J., Bailey, J., Ma, X., Gu, Q.: Improving adversarial robustness requires revisiting misclassified examples. In: International Conference on Learning Representations (2020)

2020

Later among the works it cites.

Wu, D., Wang, Y., Xia, S.t.: Revisiting loss landscape for adversarial robustness. arXiv preprint arXiv:2004.05884 (2020)

2020

Later among the works it cites.

Wu, D., Xia, S.T., Wang, Y.: Adversarial weight perturbation helps robust generalization. Advances in Neural Information Processing Systems 33

2020

Later among the works it cites.

Zhang, J., Xu, X., Han, B., Niu, G., Cui, L., Sugiyama, M., Kankanhalli, M.: Attacks which do not kill training make adversarial learning stronger. In: International conference on machine learning. pp. 11278–11287. PMLR (2020)

2020

Later among the works it cites.

Zhang, J., Zhu, J., Niu, G., Han, B., Sugiyama, M., Kankanhalli, M.: Geometry-aware instance-reweighted adversarial training. arXiv preprint arXiv:2010.01736 (2020)

2020

Later among the works it cites.

Dong, Y., Xu, K., Yang, X., Pang, T., Deng, Z., Su, H., Zhu, J.: Exploring memorization in adversarial training. arXiv preprint arXiv:2106.01606 (2021)

2021

Later among the works it cites.

Huang, T., Menkovski, V., Pei, Y., Pechenizkiy, M.: calibrated adversarial training. In: Asian Conference on Machine Learning. pp. 626–641. PMLR (2021)

2021

Later among the works it cites.

Rebuffi, S.A., Gowal, S., Calian, D.A., Stimberg, F., Wiles, O., Mann, T.: Fixing data augmentation to improve adversarial robustness. arXiv preprint arXiv:2103.01946 (2021)

2021

Later among the works it cites.

Sehwag, V., Mahloujifar, S., Handina, T., Dai, S., Xiang, C., Chiang, M., Mittal, P.: Robust learning meets generative models: Can proxy distributions improve adversarial robustness? arXiv preprint arXiv:2104.09425 (2021)

2021

Later among the works it cites.

Singla, V., Singla, S., Feizi, S., Jacobs, D.: Low curvature activations reduce overfitting in adversarial training. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 16423–16433 (2021)

2021

Later among the works it cites.

Stutz, D., Hein, M., Schiele, B.: Relating adversarially robust generalization to flat minima. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 7807–7817 (2021)

2021

Later among the works it cites.

Yu, C., Han, B., Gong, M., Shen, L., Ge, S., Du, B., Liu, T.: Robust weight perturbation for adversarial training (2021)

2021

Later among the works it cites.

Chen, T., Zhang, Z., Wang, P., Balachandra, S., Ma, H., Wang, Z., Wang, Z.: Sparsity winning twice: Better robust generaliztion from more efficient training. arXiv preprint arXiv:2202.09844 (2022)

2022

Later among the works it cites.

Huang, T., Menkovski, V., Pei, Y., Wang, Y., Pechenizkiy, M.: Direction-aggregated attack for transferable adversarial examples. ACM Journal on Emerging Technologies in Computing Systems (JETC) 18

2022

Later among the works it cites.

Pang, T., Lin, M., Yang, X., Zhu, J., Yan, S.: Robustness and accuracy could be reconcilable by (proper) definition. arXiv preprint arXiv:2202.10103 (2022)

2022

Later among the works it cites.