2021

Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models

Wang, Boxin, Xu, Chejian, Wang, Shuohang et al.

Understand

Large-scale pre-trained language models have achieved tremendous success across a wide range of natural language understanding (NLU) tasks, even surpassing human performance.

However, recent studies reveal that the robustness of these models can be challenged by carefully crafted textual adversarial examples.
While several individual datasets have been proposed to evaluate model robustness, a principled and comprehensive benchmark is still missing.
In this paper, we present Adversarial GLUE (AdvGLUE), a new multi-task benchmark to quantitatively and thoroughly explore and evaluate the vulnerabilities of modern large-scale language models under various types of adversarial attacks.

Built on

Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp
Original
J. X. Morris, E. Lifland, J. Y. Yoo, J. Grigsby, D. Jin, and Y. Qi · 2005
Earlier work this paper cites.
Distributed representations of words and phrases and their compositionality
T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean · 2013
Earlier work this paper cites.
Recursive deep models for semantic compositionality over a sentiment treebank
R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Y. Ng, and C. Potts · 2013
Earlier work this paper cites.
Glove: Global vectors for word representation
J. Pennington, R. Socher, and C. D. Manning · 2014
Earlier work this paper cites.
A large annotated corpus for learning natural language inference
S. R. Bowman, G. Angeli, C. Potts, and C. D. Manning · 2015
Earlier work this paper cites.
Explaining and harnessing adversarial examples
Original
I. J. Goodfellow, J. Shlens, and C. Szegedy · 2015
Earlier work this paper cites.
Deepfool: A simple and accurate method to fool deep neural networks
S.-M. Moosavi-Dezfooli, A. Fawzi, and P. Frossard · 2016
Earlier work this paper cites.
Distillation as a defense to adversarial perturbations against deep neural networks
N. Papernot, P. D. McDaniel, X. Wu, S. Jha, and A. Swami · 2016
Earlier work this paper cites.
Squad: 100,000+ questions for machine comprehension of text
Original
P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang · 2016
Earlier work this paper cites.
Robust physical-world attacks on deep learning models
K. Eykholt, I. Evtimov, E. Fernandes, B. Li, A. Rahmati, C. Xiao, A. Prakash, T. Kohno, and D. X. Song · 2017
Earlier work this paper cites.
Adversarial examples for evaluating reading comprehension systems
R. Jia and P. Liang · 2017
Earlier work this paper cites.
Paranmt-50m: Pushing the limits of paraphrastic sentence embeddings with millions of machine translations
Original
J. Wieting and K. Gimpel · 2017
Earlier work this paper cites.
A broad-coverage challenge corpus for sentence understanding through inference
Original
A. Williams, N. Nangia, and S. R. Bowman · 2017
Earlier work this paper cites.
Audio adversarial examples: Targeted attacks on speech-to-text
N. Carlini and D. A. Wagner · 2018
Earlier work this paper cites.
Training verified learners with learned verifiers
Original
K. Dvijotham, S. Gowal, R. Stanforth, R. Arandjelovic, B. O’Donoghue, J. Uesato, and P. Kohli · 2018
Earlier work this paper cites.
Hotflip: White-box adversarial examples for text classification
J. Ebrahimi, A. Rao, D. Lowd, and D. Dou · 2018
Earlier work this paper cites.
Datasheets for datasets
Original
T. Gebru, J. Morgenstern, B. Vecchione, J. W. Vaughan, H. Wallach, H. Daumé III, and K. Crawford · 2018
Earlier work this paper cites.
Adversarial example generation with syntactically controlled paraphrase networks
M. Iyyer, J. Wieting, K. Gimpel, and L. Zettlemoyer · 2018
Earlier work this paper cites.
Stress test evaluation for natural language inference
Original
A. Naik, A. Ravichander, N. Sadeh, C. Rose, and G. Neubig · 2018
Earlier work this paper cites.

Similar

Fever: a large-scale dataset for fact extraction and verification
J. Thorne, A. Vlachos, C. Christodoulopoulos, and A. Mittal · 2018
Cited alongside, same era.
Characterizing audio adversarial examples using temporal dependency
Original
Z. Yang, B. Li, P.-Y. Chen, and D. X. Song · 2018
Cited alongside, same era.
Certified adversarial robustness via randomized smoothing
J. M. Cohen, E. Rosenfeld, and J. Z. Kolter · 2019
Cited alongside, same era.
BERT: pre-training of deep bidirectional transformers for language understanding
J. Devlin, M. Chang, K. Lee, and K. Toutanova · 2019
Cited alongside, same era.
Achieving verified robustness to symbol substitutions via interval bound propagation
P. Huang, R. Stanforth, J. Welbl, C. Dyer, D. Yogatama, S. Gowal, K. Dvijotham, and P. Kohli · 2019
Cited alongside, same era.
Certified robustness to adversarial word substitutions
R. Jia, A. Raghunathan, K. Göksel, and P. Liang · 2019
Cited alongside, same era.
Albert: A lite bert for self-supervised learning of language representations
Original
Z.-Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut · 2019
Cited alongside, same era.
Textbugger: Generating adversarial text against real-world applications
J. Li, S. Ji, T. Du, B. Li, and T. Wang · 2019
Cited alongside, same era.
Roberta: A robustly optimized BERT pretraining approach
Original
Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov · 2019
Cited alongside, same era.
Human vs. muppet: A conservative estimate of human performance on the glue benchmark
N. Nangia and S. Bowman · 2019
Cited alongside, same era.
Openhownet: An open sememe-based lexical knowledge base
Original
F. Qi, C. Yang, Z. Liu, Q. Dong, M. Sun, and Z. Dong · 2019
Cited alongside, same era.
Adversarial attacks against fact extraction and verification
Original
J. Thorne and A. Vlachos · 2019
Cited alongside, same era.

Then

SMART: robust and efficient fine-tuning for pre-trained natural language models through principled regularized optimization
H. Jiang, P. He, W. Chen, X. Liu, J. Gao, and T. Zhao · 2020
Later among the works it cites.
Is BERT really robust? A strong baseline for natural language attack on text classification and entailment
D. Jin, Z. Jin, J. T. Zhou, and P. Szolovits · 2020
Later among the works it cites.
Bert-attack: Adversarial attack against bert using bert
L. Li, R. Ma, Q. Guo, X. Xue, and X. Qiu · 2020
Later among the works it cites.
Adversarial training for large neural language models
Original
X. Liu, H. Cheng, P. He, W. Chen, Y. Wang, H. Poon, and J. Gao · 2020
Later among the works it cites.
Reevaluating adversarial examples in natural language
J. Morris, E. Lifland, J. Lanchantin, Y. Ji, and Y. Qi · 2020
Later among the works it cites.
Adversarial NLI: A new benchmark for natural language understanding
Y. Nie, A. Williams, E. Dinan, M. Bansal, J. Weston, and D. Kiela · 2020
Later among the works it cites.
Beyond accuracy: Behavioral testing of NLP models with CheckList
M. T. Ribeiro, T. Wu, C. Guestrin, and S. Singh · 2020
Later among the works it cites.
T3: Tree-autoencoder constrained adversarial text generation for targeted attack
B. Wang, H. Pei, B. Pan, Q. Chen, S. Wang, and B. Li · 2020
Later among the works it cites.
SAFER: A structure-free approach for certified robustness to adversarial word substitutions
M. Ye, C. Gong, and Q. Liu · 2020
Later among the works it cites.
Word-level textual adversarial attacking as combinatorial optimization
Y. Zang, F. Qi, C. Yang, Z. Liu, M. Zhang, Q. Liu, and M. Sun · 2020
Later among the works it cites.
Openattack: An open-source textual adversarial attack toolkit
Original
G. Zeng, F. Qi, Q. Zhou, T. Zhang, B. Hou, Y. Zang, Z. Liu, and M. Sun · 2020
Later among the works it cites.
Freelb: Enhanced adversarial training for natural language understanding
Original
C. Zhu, Y. Cheng, Z. Gan, S. Sun, T. Goldstein, and J. Liu · 2020
Later among the works it cites.
What will it take to fix benchmarking in natural language understanding?
S. R. Bowman and G. E. Dahl · 2021
Closest in time.
Robustness gym: Unifying the nlp evaluation landscape
Original
K. Goel, N. Rajani, J. Vig, S. Tan, J. Wu, S. Zheng, C. Xiong, M. Bansal, and C. Ré · 2021
Closest in time.
Textflint: Unified multilingual robustness evaluation toolkit for natural language processing
Original
T. Gui, X. Wang, Q. Zhang, Q. Liu, Y. Zou, X. Zhou, R. Zheng, C. Zhang, Q. Wu, J. Ye, et al · 2021
Closest in time.
Dynabench: Rethinking benchmarking in nlp
D. Kiela, M. Bartolo, Y. Nie, D. Kaushik, A. Geiger, Z. Wu, B. Vidgen, G. Prasad, A. Singh, P. Ringshia, Z. Ma, T. Thrush, S. Riedel, Z. Waseem, P. Stenetorp, R. Jia, M. Bansal, C. Potts, and A. Williams · 2021
Closest in time.
Left, right, and gender: Exploring interaction traces to mitigate human biases
Original
E. Wall, A. Narechania, A. Coscia, J. Paden, and A. Endert · 2021
Closest in time.
Infobert: Improving robustness of language models from an information theoretic perspective
B. Wang, S. Wang, Y. Cheng, Z. Gan, R. Jia, B. Li, and J. Liu · 2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…