2021

ShufText: A Simple Black Box Approach to Evaluate the Fragility of Text Classification Models

Taware, Rutuja, Varat, Shraddha, Salunke, Gaurav et al.

Understand

Text classification is the most basic natural language processing task.

It has a wide range of applications ranging from sentiment analysis to topic classification.
Recently, deep learning approaches based on CNN, LSTM, and Transformers have been the de facto approach for text classification.
In this work, we highlight a common issue associated with these approaches.

Built on

Li, X., Roth, D.: Learning question classifiers. In: COLING 2002: The 19th International Conference on Computational Linguistics (2002), https://www.aclweb.org/anthology/C02-1150
2002
Earlier work this paper cites.
Ebrahimi, J., Rao, A., Lowd, D., Dou, D.: HotFlip: White-box adversarial examples for text classification. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). pp. 31–36. Association for Computational Linguistics, Melbourne, Australia (Jul 2018). https://doi.org/10.18653/v1/P18-2006, https://www.aclweb.org/anthology/P18-2006
2006
Earlier work this paper cites.
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C.D., Ng, A., Potts, C.: Recursive deep models for semantic compositionality over a sentiment treebank. In: Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. pp. 1631–1642. Association for Computational Linguistics, Seattle, Washington, USA (Oct 2013), https://www.aclweb.org/anthology/D13-1170
2013
Earlier work this paper cites.
Kim, Y.: Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882 (2014)
Original
2014
Earlier work this paper cites.
Nguyen, A., Yosinski, J., Clune, J.: Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 427–436 (2015)
2015
Earlier work this paper cites.
Rios, A., Kavuluru, R.: Convolutional neural networks for biomedical text classification: application in indexing biomedical articles. In: Proceedings of the 6th ACM Conference on Bioinformatics, Computational Biology and Health Informatics. pp. 258–267 (2015)
2015
Earlier work this paper cites.
Zhou, C., Sun, C., Liu, Z., Lau, F.: A c-lstm neural network for text classification. arXiv preprint arXiv:1511.08630 (2015)
Original
2015
Earlier work this paper cites.
Li, J., Monroe, W., Jurafsky, D.: Understanding neural networks through representation erasure. arXiv preprint arXiv:1612.08220 (2016)
Original
2016
Earlier work this paper cites.
Liu, B., Lane, I.: Attention-based recurrent neural network models for joint intent detection and slot filling. arXiv preprint arXiv:1609.01454 (2016)
Original
2016
Earlier work this paper cites.

Similar

Ribeiro, M.T., Singh, S., Guestrin, C.: ” why should i trust you?” explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. pp. 1135–1144 (2016)
2016
Cited alongside, same era.
Liang, B., Li, H., Su, M., Bian, P., Li, X., Shi, W.: Deep text classification can be fooled. CoRR abs/1704.08006
Original
2017
Cited alongside, same era.
Conneau, A., Kruszewski, G., Lample, G., Barrault, L., Baroni, M.: What you can cram into a single vector: Probing sentence embeddings for linguistic properties. arXiv preprint arXiv:1805.01070 (2018)
Original
2018
Cited alongside, same era.
Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. CoRR abs/1810.04805
Original
2018
Cited alongside, same era.
Feng, S., Wallace, E., Grissom II, A., Iyyer, M., Rodriguez, P., Boyd-Graber, J.: Pathologies of neural models make interpretations difficult. arXiv preprint arXiv:1804.07781 (2018)
Original
2018
Cited alongside, same era.
Gao, J., Lanchantin, J., Soffa, M.L., Qi, Y.: Black-box generation of adversarial text sequences to evade deep learning classifiers. CoRR abs/1801.04354
Original
2018
Cited alongside, same era.
Howard, J., Ruder, S.: Fine-tuned language models for text classification. CoRR abs/1801.06146
Original
2018
Cited alongside, same era.
Jacovi, A., Shalom, O.S., Goldberg, Y.: Understanding convolutional neural networks for text classification. arXiv preprint arXiv:1809.08037 (2018)
Original
2018
Cited alongside, same era.
Ribeiro, M.T., Singh, S., Guestrin, C.: Semantically equivalent adversarial rules for debugging nlp models. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 856–865 (2018)
2018
Cited alongside, same era.

Then

DeYoung, J., Jain, S., Rajani, N.F., Lehman, E., Xiong, C., Socher, R., Wallace, B.C.: Eraser: A benchmark to evaluate rationalized nlp models. arXiv preprint arXiv:1911.03429 (2019)
Original
2019
Later among the works it cites.
Joshi, R., Goel, P., Joshi, R.: Deep learning for hindi text classification: A comparison. In: International Conference on Intelligent Human Computer Interaction. pp. 94–101. Springer (2019)
2019
Later among the works it cites.
Kowsari, K., Jafari Meimandi, K., Heidarysafa, M., Mendu, S., Barnes, L., Brown, D.: Text classification algorithms: A survey. Information 10
2019
Later among the works it cites.
Wallace, E., Tuyls, J., Wang, J., Subramanian, S., Gardner, M., Singh, S.: Allennlp interpret: A framework for explaining predictions of nlp models. arXiv preprint arXiv:1909.09251 (2019)
Original
2019
Later among the works it cites.
Yuan, H., Chen, Y., Hu, X., Ji, S.: Interpreting deep models for text analysis via optimization and regularization methods. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 33, pp. 5717–5724 (2019)
2019
Later among the works it cites.
Desai, U., Tamilselvam, S., Kaur, J., Mani, S., Khare, S.: Benchmarking popular classification models’ robustness to random and targeted corruptions (2020)
2020
Later among the works it cites.
NA: Amazon confuses sarcastic tweet on maharashtra turmoil for customer complaint, deletes it later. https://www.news18.com/news/buzz/amazon-help-confuses-sarcastic-tweet-on-maharashtra- political-crisis-for-customer-complaint-deletes-tweet-later-2399797.html , (Accessed on 12/31/2020)
2020
Later among the works it cites.
Ribeiro, M.T., Wu, T., Guestrin, C., Singh, S.: Beyond accuracy: Behavioral testing of nlp models with checklist. arXiv preprint arXiv:2005.04118 (2020)
Original
2020
Later among the works it cites.
Kulkarni, A., Mandhane, M., Likhitkar, M., Kshirsagar, G., Jagdale, J., Joshi, R.: Experimental evaluation of deep learning models for marathi text classification. arXiv preprint arXiv:2101.04899 (2021)
Original
2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…