Fetching the paper…

Reading the bibliography…

2022

Plug and Play Counterfactual Text Generation for Model Robustness

Madaan, Nishtha, Bedathur, Srikanta, Saha, Diptikalyan

Original
PDF
Post on X

Understand

Generating counterfactual test-cases is an important backbone for testing NLP models and making them as robust and reliable as traditional software.

In generating the test-cases, a desired property is the ability to control the test-case generation in a flexible manner to test for a large variety of failure cases and to explain and repair them in a targeted manner.
In this direction, significant progress has been made in the prior works by manually writing rules for generating controlled counterfactuals.
However, this approach requires heavy manual supervision and lacks the flexibility to easily introduce new controls.

Plug and Play Counterfactual Text Generation for Model Robustness · Around

Built on

On evaluation of adversarial perturbations for sequence-to-sequence models
Original
Paul Michel, Xian Li, Graham Neubig, and Juan Miguel Pino. 2019 · 1903
Earlier work this paper cites.
Roberta: A robustly optimized bert pretraining approach
Original
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019 · 1907
Earlier work this paper cites.
Can unconditional language models recover arbitrary sentences?
Original
Nishant Subramani, Samuel R Bowman, and Kyunghyun Cho. 2019 · 1907
Earlier work this paper cites.
Ctrl: A conditional transformer language model for controllable generation
Original
Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. 2019 · 1909
Earlier work this paper cites.
Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension
Original
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019 · 1910
Earlier work this paper cites.
Plug and play language models: a simple approach to controlled text generation
Original
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. 2019 · 1912
Earlier work this paper cites.
Models, reasoning and inference
Judea Pearl et al. 2000 · 2000
Earlier work this paper cites.
Variational template machine for data-to-text generation
Original
Rong Ye, Wenxian Shi, Hao Zhou, Zhongyu Wei, and Lei Li. 2020 · 2002
Earlier work this paper cites.
Beyond accuracy: Behavioral testing of nlp models with checklist
Original
Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020 · 2005
Earlier work this paper cites.
Gedi: Generative discriminator guided sequence generation
Original
Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. 2020 · 2009
Earlier work this paper cites.
Explaining the efficacy of counterfactually-augmented data
Original
Divyansh Kaushik, Amrith Setlur, Eduard Hovy, and Zachary C Lipton. 2020 · 2010
Earlier work this paper cites.
Cat-gen: Improving robustness in nlp models via controlled adversarial text generation
Original
Tianlu Wang, Xuezhi Wang, Yao Qin, Ben Packer, Kang Li, Jilin Chen, Alex Beutel, and Ed Chi. 2020 · 2010
Earlier work this paper cites.
Learning word vectors for sentiment analysis
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011 · 2011
Earlier work this paper cites.
Explaining nlp models via minimal contrastive editing (mice)
Original
Alexis Ross, Ana Marasović, and Matthew E Peters. 2020 · 2012
Earlier work this paper cites.
Explaining and harnessing adversarial examples
Original
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. 2014 · 2014
Earlier work this paper cites.
Character-level convolutional networks for text classification
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015 · 2015
Earlier work this paper cites.
Learning to translate in real-time with neural machine translation
Original
Jiatao Gu, Graham Neubig, Kyunghyun Cho, and Victor OK Li. 2016 · 2016
Earlier work this paper cites.
Understanding neural networks through representation erasure
Original
Jiwei Li, Will Monroe, and Dan Jurafsky. 2016 · 2016
Earlier work this paper cites.

Similar

Hotflip: White-box adversarial examples for text classification
Original
Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. 2017 · 2017
Cited alongside, same era.
Fairness testing: testing software for discrimination
Sainyam Galhotra, Yuriy Brun, and Alexandra Meliou. 2017 · 2017
Cited alongside, same era.
Trainable greedy decoding for neural machine translation
Original
Jiatao Gu, Kyunghyun Cho, and Victor OK Li. 2017 · 2017
Cited alongside, same era.
Toward controlled generation of text
Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P Xing. 2017 · 2017
Cited alongside, same era.
Adversarial examples for evaluating reading comprehension systems
Original
Robin Jia and Percy Liang. 2017 · 2017
Cited alongside, same era.

Then

Learning to control the fine-grained sentiment for story ending generation
Fuli Luo, Damai Dai, Pengcheng Yang, Tianyu Liu, Baobao Chang, Zhifang Sui, and Xu Sun. 2019 · 2019
Later among the works it cites.
Language models are unsupervised multitask learners
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019 · 2019
Later among the works it cites.
Sentence-bert: Sentence embeddings using siamese bert-networks
Original
Nils Reimers and Iryna Gurevych. 2019 · 2019
Later among the works it cites.
Errudite: Scalable, reproducible, and testable error analysis
Tongshuang Wu, Marco Tulio Ribeiro, Jeffrey Heer, and Daniel Weld. 2019 · 2019
Later among the works it cites.
Good-enough compositional data augmentation
Jacob Andreas. 2020 · 2020
Later among the works it cites.
Evaluating models’ local decision boundaries via contrast sets

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Plug & play generative networks: Conditional iterative generation of images in latent space

Anh Nguyen, Jeff Clune, Yoshua Bengio, Alexey Dosovitskiy, and Jason Yosinski. 2017 · 2017

Cited alongside, same era.

Style transfer from non-parallel text by cross-alignment

Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2017 · 2017

Cited alongside, same era.

Counterfactual explanations without opening the black box: Automated decisions and the gdpr

Sandra Wachter, Brent Mittelstadt, and Chris Russell. 2017 · 2017

Cited alongside, same era.

A stable and effective learning strategy for trainable greedy decoding

Yun Chen, Victor OK Li, Kyunghyun Cho, and Samuel R Bowman. 2018 · 2018

Cited alongside, same era.

Bert: Pre-training of deep bidirectional transformers for language understanding

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018 · 2018

Cited alongside, same era.

Adversarial example generation with syntactically controlled paraphrase networks

Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. 2018 · 2018

Cited alongside, same era.

Multiple-attribute text rewriting

Guillaume Lample, Sandeep Subramanian, Eric Smith, Ludovic Denoyer, Marc’Aurelio Ranzato, and Y-Lan Boureau. 2018 · 2018

Cited alongside, same era.

Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hannaneh Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, and Ben Zhou. 2020 · 2020

Later among the works it cites.

Verifying individual fairness in machine learning models

Philips George John, Deepak Vijaykeerthy, and Diptikalyan Saha. 2020 · 2020

Later among the works it cites.

BERT-ATTACK: Adversarial attack against BERT using BERT

Linyang Li, Ruotian Ma, Qipeng Guo, Xiangyang Xue, and Xipeng Qiu. 2020b · 2020

Later among the works it cites.

Metamorphic testing and certified mitigation of fairness violations in nlp models

Pingchuan Ma, Shuai Wang, and Jin Liu. 2020 · 2020

Later among the works it cites.

Politeness transfer: A tag and generate approach

Aman Madaan, Amrith Setlur, Tanmay Parekh, Barnabas Poczos, Graham Neubig, Yiming Yang, Ruslan Salakhutdinov, Alan W Black, and Shrimai Prabhumoye. 2020 · 2020

Later among the works it cites.

Unsupervised text style transfer with padded masked language models

Eric Malmi, Aliaksei Severyn, and Sascha Rothe. 2020 · 2020

Later among the works it cites.

Explaining machine learning classifiers through diverse counterfactual explanations

Ramaravind K Mothilal, Amit Sharma, and Chenhao Tan. 2020 · 2020

Later among the works it cites.

Learning what makes a difference from counterfactual examples and gradient supervision

Damien Teney, Ehsan Abbasnedjad, and Anton van den Hengel. 2020 · 2020

Later among the works it cites.

Generate your counterfactuals: Towards controlled counterfactual generation for text

Nishtha Madaan, Inkit Padhi, Naveen Panwar, and Diptikalyan Saha. 2021 · 2021

Later among the works it cites.

Lewis: Levenshtein editing for unsupervised text style transfer

Machel Reid and Victor Zhong. 2021 · 2021

Later among the works it cites.

Tailor: Generating and perturbing text with semantic controls

Alexis Ross, Tongshuang Wu, Hao Peng, Matthew E. Peters, and Matt Gardner. 2021 · 2021

Later among the works it cites.

Polyjuice: Automated, general-purpose counterfactual generation

Tongshuang Wu, Marco Tulio Ribeiro, Jeffrey Heer, and Daniel S Weld. 2021 · 2021

Later among the works it cites.