2020

To BAN or not to BAN: Bayesian Attention Networks for Reliable Hate Speech Detection

Miok, Kristian, Skrlj, Blaz, Zaharie, Daniela et al.

Understand

Hate speech is an important problem in the management of user-generated content.

To remove offensive content or ban misbehaving users, content moderators need reliable hate speech detectors.
Recently, deep neural networks based on the transformer architecture, such as the (multilingual) BERT model, achieve superior performance in many natural language classification tasks, including hate speech detection.
So far, these methods have not been able to quantify their output in terms of reliability.

Built on

Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R (2014) Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research 15(1):1929–1958
1958
Earlier work this paper cites.
Plutchik R (1982) Emotion, a psychoevolutionary synthesis. Harper & Row, New York
1982
Earlier work this paper cites.
Platt JC (1999) Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In: Advances in large margin classifiers, MIT Press, pp 61–74
1999
Earlier work this paper cites.
Breiman L (2001) Random forests. Machine Learning Journal 45:5–32
2001
Earlier work this paper cites.
Zadrozny B, Elkan C (2002) Transforming classifier scores into accurate multiclass probability estimates. In: Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, pp 694–699
2002
Earlier work this paper cites.
Niculescu-Mizil A, Caruana R (2005) Predicting good probabilities with supervised learning. In: Proceedings of the 22nd international conference on Machine learning, pp 625–632
2005
Earlier work this paper cites.
Cambria E, Chandra P, Sharma A, Hussain A (2010) Do not feel the trolls. ISWC, Shanghai
2010
Earlier work this paper cites.
Bleich E (2011) The rise of hate speech and hate crime laws in liberal democracies. Journal of Ethnic and Migration Studies 37(6):917–934
2011
Earlier work this paper cites.
Cambria E, Livingstone A, Hussain A (2012) The hourglass of emotions. In: Cognitive behavioural systems, Springer, pp 144–157
2012
Earlier work this paper cites.
Warner W, Hirschberg J (2012) Detecting hate speech on the world wide web. In: Proceedings of the second workshop on language in social media, pp 19–26
2012
Earlier work this paper cites.
Mikolov T, Sutskever I, Chen K, Corrado GS, Dean J (2013) Distributed representations of words and phrases and their compositionality. In: Advances in neural information processing systems, pp 3111–3119
2013
Earlier work this paper cites.
Wang S, Manning C (2013) Fast dropout training. In: International Conference on Machine Learning, pp 118–126
2013
Earlier work this paper cites.
Kingma DP, Ba J (2014) Adam: A method for stochastic optimization. arXiv preprint arXiv:14126980
2014
Earlier work this paper cites.
Zaremba W, Sutskever I, Vinyals O (2014) Recurrent neural network regularization. arXiv preprint arXiv:14092329
2014
Earlier work this paper cites.
Bluche T, Kermorvant C, Louradour J (2015) Where to apply dropout in recurrent neural networks for handwriting recognition? In: 2015 13th International Conference on Document Analysis and Recognition (ICDAR), pp 681–685
2015
Earlier work this paper cites.
Cambria E, Hussain A (2015) Sentic Computing: A Common-Sense-Based Framework for Concept-Level Sentiment Analysis. Springer, Cham, Switzerland
2015
Earlier work this paper cites.
McAuley J, Targett C, Shi Q, Van Den Hengel A (2015) Image-based recommendations on styles and substitutes. In: Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp 43–52
2015
Earlier work this paper cites.
Cambria E (2016) Affective computing and sentiment analysis. IEEE intelligent systems 31(2):102–107
2016
Earlier work this paper cites.
Cambria E, Poria S, Bajpai R, Schuller B (2016) Senticnet 4: A semantic resource for sentiment analysis based on conceptual primitives. In: Proceedings of COLING 2016, the 26th international conference on computational linguistics: Technical papers, pp 2666–2677
2016
Earlier work this paper cites.
Gal Y (2016) Uncertainty in deep learning. University of Cambridge 1:3
2016
Earlier work this paper cites.
Mehdad Y, Tetreault J (2016) Do characters abuse more than words? In: Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pp 299–303
2016
Earlier work this paper cites.
Myshkov P, Julier S (2016) Posterior distribution analysis for Bayesian inference in neural networks. In: Workshop on Bayesian Deep Learning, NIPS 2016, Barcelona, Spain
2016
Earlier work this paper cites.

Similar

Waseem Z, Hovy D (2016) Hateful symbols or hateful people? Predictive features for hate speech detection on Twitter. In: Proceedings of the NAACL student research workshop, pp 88–93
2016
Cited alongside, same era.
Yang Z, Yang D, Dyer C, He X, Smola A, Hovy E (2016) Hierarchical attention networks for document classification. In: Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies, pp 1480–1489
2016
Cited alongside, same era.
Davidson T, Warmsley D, Macy M, Weber I (2017) Automated hate speech detection and the problem of offensive language. In: Proceedings of the 11th International AAAI Conference on Web and Social Media, ICWSM ’17, pp 512–515
2017
Cited alongside, same era.
Du J, Xu R, He Y, Gui L (2017) Stance classification with target-specific neural attention networks. In: Proceedings of International Joint Conferences on Artificial Intelligence, IJCAI 2017, pp 3988–3994
2017
Cited alongside, same era.
Fišer D, Erjavec T, Ljubešić N (2017) Legal framework, dataset and annotation schema for socially unacceptable online discourse practices in Slovene. In: Proceedings of the first workshop on abusive language online, pp 46–51
2017
Cited alongside, same era.
Fortunato M, Blundell C, Vinyals O (2017) Bayesian recurrent neural networks. arXiv preprint arXiv:170402798
2017
Cited alongside, same era.
Guo C, Pleiss G, Sun Y, Weinberger KQ (2017) On calibration of modern neural networks. In: Proceedings of the 34th International Conference on Machine Learning, pp 1321–1330
2017
Cited alongside, same era.
Lakshminarayanan B, Pritzel A, Blundell C (2017) Simple and scalable predictive uncertainty estimation using deep ensembles. In: Advances in neural information processing systems, pp 6402–6413
2017
Cited alongside, same era.
Malmasi S, Zampieri M (2017) Detecting hate speech in social media. In: Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP 2017, Varna, Bulgaria, pp 467–472
2017
Cited alongside, same era.
Pappas N, Popescu-Belis A (2017) Multilingual hierarchical attention networks for document classification. In: 8th International Joint Conference on Natural Language Processing, pp 1015–1025
2017
Cited alongside, same era.
Schmidt A, Wiegand M (2017) A survey on hate speech detection using natural language processing. In: Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media, pp 1–10
2017
Cited alongside, same era.
Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I (2017) Attention is all you need. In: Advances in neural information processing systems, pp 5998–6008
2017
Cited alongside, same era.

Then

Vezjak B (2018) Radical hate speech: The fascination with Hitler and fascism on the Slovenian webosphere. Solsko Polje 29
2018
Later among the works it cites.
Bauwelinck N, Lefever E (2019) Measuring the impact of sentiment for hate speech detection on Twitter. In: The Fifth International Conference on Human and Social Analytics (HUSO), pp 17–22
2019
Later among the works it cites.
Chang WC, Yu HF, Zhong K, Yang Y, Dhillon I (2019) X-BERT: eXtreme multi-label text classification with BERT. arXiv preprint arXiv:190502331
2019
Later among the works it cites.
Devlin J, Chang MW, Lee K, Toutanova K (2019) BERT: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp 4171–4186
2019
Later among the works it cites.
Gururangan S, Dang T, Card D, Smith NA (2019) Variational pretraining for semi-supervised text classification. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp 5880–5894
2019
Later among the works it cites.
Kocijan K, Košković L, Bajac P (2019) Detecting hate speech online: A case of Croatian. In: International Conference on Automatic Processing of Natural-Language Electronic Texts with NooJ, pp 185–197
2019
Later among the works it cites.
Ljubešić N, Fišer D, Erjavec T (2019) The FRENK datasets of socially unacceptable discourse in Slovene and English. In: International Conference on Text, Speech, and Dialogue, Springer, pp 103–114
2019
Later among the works it cites.
Maddox WJ, Izmailov P, Garipov T, Vetrov DP, Wilson AG (2019) A simple baseline for Bayesian uncertainty in deep learning. In: Advances in Neural Information Processing Systems, pp 13132–13143
2019
Later among the works it cites.
Marinšek R (2019) Cross-lingual embeddings for hate speech detection in comments. MSc thesis, University of Ljubljana, Faculty of Computer and Information Science
2019
Later among the works it cites.
Miok K, Nguyen-Doan D, Robnik-Šikonja M, Zaharie D (2019a) Multiple imputation for biomedical data using Monte Carlo dropout autoencoders. 2019 E-Health and Bioengineering Conference (EHB) pp 1–4
2019
Later among the works it cites.
Miok K, Nguyen-Doan D, Zaharie D, Robnik-Šikonja M (2019c) Generating data using Monte Carlo dropout. In: 2019 IEEE 15th International Conference on Intelligent Computer Communication and Processing (ICCP), IEEE, pp 509–515
2019
Later among the works it cites.
Mozafari M, Farahbakhsh R, Crespi N (2019) A BERT-based transfer learning approach for hate speech detection in online social media. In: International Conference on Complex Networks and Their Applications, Springer, pp 928–940
2019
Later among the works it cites.
Pamungkas EW, Patti V (2019) Cross-domain and cross-lingual abusive language detection: A hybrid approach with deep learning and a multilingual lexicon. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, pp 363–370
2019
Later among the works it cites.
Rodríguez A, Argueta C, Chen YL (2019) Automatic detection of hate speech on Facebook using sentiment and emotion analysis. In: 2019 International Conference on Artificial Intelligence in Information and Communication (ICAIIC), pp 169–174
2019
Later among the works it cites.
Sun C, Qiu X, Xu Y, Huang X (2019) How to fine-tune BERT for text classification? In: China National Conference on Chinese Computational Linguistics, pp 194–206
2019
Later among the works it cites.
Cambria E, Li Y, Xing FZ, Poria S, Kwok K (2020) Senticnet 6: Ensemble application of symbolic and subsymbolic ai for sentiment analysis. In: Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pp 105–114
2020
Closest in time.
Izmailov P, Maddox WJ, Kirichenko P, Garipov T, Vetrov D, Wilson AG (2020) Subspace inference for Bayesian deep learning. In: Uncertainty in Artificial Intelligence, PMLR, pp 1169–1179
2020
Closest in time.
Rogers A, Kovaleva O, Rumshisky A (2020) A primer in BERTology: What we know about how BERT works. arXiv preprint arXiv:200212327
2020
Closest in time.
Stappen L, Brunn F, Schuller B (2020) Cross-lingual zero-and few-shot hate speech detection utilising frozen transformer language models and AXEL. arXiv preprint arXiv:200413850
2020
Closest in time.
Susanto Y, Livingstone AG, Ng BC, Cambria E (2020) The hourglass model revisited. IEEE Intelligent Systems 35(5):96–102
2020
Closest in time.
Wiedemann G, Yimam SM, Biemann C (2020) UHH-LT & LT2 at SemEval-2020 Task 12: Fine-tuning of pre-trained transformer networks for offensive language detection. arXiv preprint arXiv:200411493
2020
Closest in time.
Xu Y, Qiu X, Zhou L, Huang X (2020) Improving BERT fine-tuning via self-ensemble and self-distillation. arXiv preprint arXiv:200210345
2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…