2021

HeBERT & HebEMO: a Hebrew BERT Model and a Tool for Polarity Analysis and Emotion Recognition

Chriqui, Avihay, Yahav, Inbal

Understand

This paper introduces HeBERT and HebEMO.

HeBERT is a Transformer-based model for modern Hebrew text, which relies on a BERT (Bidirectional Encoder Representations for Transformers) architecture.
BERT has been shown to outperform alternative architectures in sentiment analysis, and is suggested to be particularly appropriate for MRLs.
Analyzing multiple BERT specifications, we find that while model complexity correlates with high performance on language tasks that aim to understand terms in a sentence, a more-parsimonious model better captures the sentiment of entire sentence.

Built on

Zampieri M, Malmasi S, Nakov P, Rosenthal S, Farra N, Kumar R (2019) Semeval-2019 task 6: Identifying and categorizing offensive language in social media (offenseval). arXiv preprint arXiv:1903.08983
Original
1903
Earlier work this paper cites.
Zhong P, Wang D, Miao C (2019) Knowledge-enriched transformer for emotion detection in textual conversations. arXiv preprint arXiv:1909.10681
Original
1909
Earlier work this paper cites.
Krippendorff K (1970) Estimating the reliability, systematic error and random error of interval data. Educational and Psychological Measurement 30(1):61–70
1970
Earlier work this paper cites.
Plutchik R (1980) A general psychoevolutionary theory of emotion. Theories of emotion , 3–33 (Elsevier)
1980
Earlier work this paper cites.
Rosaldo MZ, Shweder RA, LeVine RA (1984) Culture theory: essays on mind, self, and emotion
1984
Earlier work this paper cites.
Ortony A, Clore GL, Foss MA (1987) The referential structure of the affective lexicon. Cognitive science 11(3):341–364
1987
Earlier work this paper cites.
Wierzbicka A (1994) Emotion, language, and cultural scripts.
1994
Earlier work this paper cites.
Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural computation 9(8):1735–1780
1997
Earlier work this paper cites.
Ekman P (1999) Basic emotions. Handbook of cognition and emotion 98(45-60):16
1999
Earlier work this paper cites.
Pennebaker JW, Francis ME, Booth RJ (2001) Linguistic inquiry and word count: Liwc 2001. Mahway: Lawrence Erlbaum Associates 71(2001):2001
2001
Earlier work this paper cites.
Sima’an K, Itai A, Winter Y, Altman A, Nativ N (2001) Building a tree-bank of modern hebrew text. Traitement Automatique des Langues 42(2):247–380
2001
Earlier work this paper cites.
Antoun W, Baly F, Hajj H (2020) Arabert: Transformer-based model for arabic language understanding. arXiv preprint arXiv:2003.00104
Original
2003
Earlier work this paper cites.
Kövecses Z (2003) Metaphor and emotion: Language, culture, and body in human feeling (Cambridge University Press)
2003
Earlier work this paper cites.
Mordecai NB, Elhadad M (2005) Hebrew named entity recognition. MONEY 81(83.93):82–49
2005
Earlier work this paper cites.
Tsarfaty R, Bareket D, Klein S, Seker A (2020) From spmrl to nmrl: What did we learn (and unlearn) in a decade of parsing morphologically-rich languages (mrls)? arXiv preprint arXiv:2005.01330
Original
2005
Earlier work this paper cites.
Bareket D, Tsarfaty R (2020) Neural modeling for named entities and morphology (nemoˆ 2). arXiv preprint arXiv:2007.15620
Original
2007
Earlier work this paper cites.
Chitturi R, Raghunathan R, Mahajan V (2007) Form versus function: How the intensities of specific emotions evoked in functional versus hedonic trade-offs mediate product preferences. Journal of marketing research 44(4):702–714
2007
Earlier work this paper cites.
Patwa P, Aguilar G, Kar S, Pandey S, PYKL S, Gambäck B, Chakraborty T, Solorio T, Das A (2020) Semeval-2020 task 9: Overview of sentiment analysis of code-mixed tweets. arXiv e-prints arXiv–2008
2008
Earlier work this paper cites.
Fattah K, Fierke KM (2009) A clash of emotions: The politics of humiliation and political violence in the middle east. European journal of international relations 15(1):67–93
2009
Earlier work this paper cites.
Kim-Prieto C, Diener E (2009) Religion as a source of variation in the experience of positive and negative emotions. The Journal of Positive Psychology 4(6):447–460
2009
Earlier work this paper cites.
Pan SJ, Yang Q (2009) A survey on transfer learning. IEEE Transactions on knowledge and data engineering 22(10):1345–1359
2009
Earlier work this paper cites.
Tay Y, Dehghani M, Bahri D, Metzler D (2020) Efficient transformers: A survey. arXiv preprint arXiv:2009.06732
Original
2009
Earlier work this paper cites.
Argaman O (2010) Linguistic markers and emotional intensity. Journal of psycholinguistic research 39(2):89–99
2010
Earlier work this paper cites.
Levine Y, Lenz B, Lieber O, Abend O, Leyton-Brown K, Tennenholtz M, Shoham Y (2020) Pmi-masking: Principled masking of correlated spans. arXiv preprint arXiv:2010.01825
Original
2010
Earlier work this paper cites.
Tsarfaty R, Seddah D, Goldberg Y, Kübler S, Versley Y, Candito M, Foster J, Rehbein I, Tounsi L (2010) Statistical parsing of morphologically rich languages (spmrl) what, how and whither. Proceedings of the NAACL HLT 2010 First Workshop on Statistical Parsing of Morphologically-Rich Languages , 1–12
2010
Earlier work this paper cites.

Similar

Li S, Ju S, Zhou G, Lin X (2012) Active learning for imbalanced sentiment classification. Proceedings of the 2012 Joint conference on empirical methods in natural language processing and computational natural language learning , 139–148
2012
Cited alongside, same era.
Liu B (2012) Sentiment analysis and opinion mining. Synthesis lectures on human language technologies 5(1):1–167
2012
Cited alongside, same era.
Liu B, Zhang L (2012) A survey of opinion mining and sentiment analysis. Mining text data , 415–463 (Springer)
2012
Cited alongside, same era.
Schuster M, Nakajima K (2012) Japanese and korean voice search. 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 5149–5152 (IEEE)
2012
Cited alongside, same era.
Desmet B, Hoste V (2013) Emotion detection in suicide notes. Expert Systems with Applications 40(16):6351–6358
2013
Cited alongside, same era.
Mohammad SM, Turney PD (2013) Crowdsourcing a word-emotion association lexicon 29(3):436–465
2013
Cited alongside, same era.
Dong L, Wei F, Tan C, Tang D, Zhou M, Xu K (2014) Adaptive recursive neural network for target-dependent twitter sentiment classification. Proceedings of the 52nd annual meeting of the association for computational linguistics (volume 2: Short papers) , 49–54
2014
Cited alongside, same era.
Kim Y (2014) Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882
Original
2014
Cited alongside, same era.
Medhat W, Hassan A, Korashy H (2014) Sentiment analysis algorithms and applications: A survey. Ain Shams engineering journal 5(4):1093–1113
2014
Cited alongside, same era.
Bojanowski P, Joulin A, Mikolov T (2015) Alternative structures for character-level rnns. arXiv preprint arXiv:1511.06303
Original
2015
Cited alongside, same era.
Joulin A, Grave E, Bojanowski P, Douze M, Jégou H, Mikolov T (2016) Fasttext.zip: Compressing text classification models. arXiv preprint arXiv:1612.03651
Original
2016
Cited alongside, same era.
Steiner T (2016) President rivlin’s" four tribes" initiative: The foreign policy implications of a democratic & inclusive process to address israel’s socio-demographic transformation
2016
Cited alongside, same era.

Then

Radford A, Narasimhan K, Salimans T, Sutskever I (2018) Improving language understanding by generative pre-training
2018
Later among the works it cites.
Vania C, Grivas A, Lopez A (2018) What do character-level models learn about morphology? the case of dependency parsing. arXiv preprint arXiv:1808.09180
Original
2018
Later among the works it cites.
Zhang L, Wang S, Liu B (2018) Deep learning for sentiment analysis: A survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 8(4):e1253
2018
Later among the works it cites.
Chatterjee A, Narahari KN, Joshi M, Agrawal P (2019) Semeval-2019 task 3: Emocontext contextual emotion detection in text. Proceedings of the 13th International Workshop on Semantic Evaluation , 39–48
2019
Later among the works it cites.
Ghanbari-Adivi F, Mosleh M (2019) Text emotion detection in social networks using a novel ensemble classifier based on parzen tree estimator (tpe). Neural Computing and Applications 31(12):8971–8983
2019
Later among the works it cites.
Hemmatian F, Sohrabi MK (2019) A survey on classification techniques for opinion mining and sentiment analysis. Artificial Intelligence Review 1–51
2019
Later among the works it cites.
Jawahar G, Sagot B, Seddah D (2019) What does bert learn about the structure of language?
2019
Later among the works it cites.
Liu R, Shi Y, Ji C, Jia M (2019) A survey of sentiment analysis based on transfer learning. IEEE Access 7:85401–85412
2019
Later among the works it cites.
More A, Seker A, Basmova V, Tsarfaty R (2019) Joint transition-based models for morpho-syntactic parsing: Parsing strategies for mrls and a case study from modern hebrew. Transactions of the Association for Computational Linguistics 7:33–48
2019
Later among the works it cites.
Pota M, Marulli F, Esposito M, De Pietro G, Fujita H (2019) Multilingual pos tagging by a composite deep architecture based on character-level features and on-the-fly enriched word embeddings. Knowledge-Based Systems 164:309–323
2019
Later among the works it cites.
Radford A, Wu J, Child R, Luan D, Amodei D, Sutskever I (2019) Language models are unsupervised multitask learners. OpenAI blog 1(8):9
2019
Later among the works it cites.
Yue L, Chen W, Li X, Zuo W, Yin M (2019) A survey of sentiment analysis in social media. Knowledge and Information Systems 1–47
2019
Later among the works it cites.
Acheampong FA, Wenyu C, Nunoo-Mensah H (2020) Text-based emotion detection: Advances, challenges, and opportunities. Engineering Reports e12189
2020
Later among the works it cites.
Ahmad Z, Jindal R, Ekbal A, Bhattachharyya P (2020) Borrow from rich cousin: transfer learning for emotion detection using cross lingual embedding. Expert Systems with Applications 139:112851
2020
Later among the works it cites.
Ahorsu DK, Lin CY, Imani V, Saffari M, Griffiths MD, Pakpour AH (2020) The fear of covid-19 scale: development and initial validation. International journal of mental health and addiction
2020
Later among the works it cites.
Bellstam G, Bhagat S, Cookson JA (2020) A text-based analysis of corporate innovation. Management Science
2020
Later among the works it cites.
Klein S, Tsarfaty R (2020) Getting the## life out of living: How adequate are word-pieces for modelling complex morphology? Proceedings of the 17th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology , 204–209
2020
Later among the works it cites.
Ortiz Suárez PJ, Romary L, Sagot B (2020) A monolingual approach to contextualized word embeddings for mid-resource languages. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , 1703–1714 (Online: Association for Computational Linguistics), URL https://www.aclweb.org/anthology/2020.acl-main.156
2020
Later among the works it cites.
Pedrosa AL, Bitencourt L, Fróes ACF, Cazumbá MLB, Campos RGB, de Brito SBCS, e Silva ACS (2020) Emotional, behavioral, and psychological impact of the covid-19 pandemic. Frontiers in psychology 11
2020
Later among the works it cites.
Pfefferbaum B, North CS (2020) Mental health and the covid-19 pandemic. New England Journal of Medicine
2020
Later among the works it cites.
Shapira N, Lazarus G, Goldberg Y, Gilboa-Schechtman E, Tuval-Mashiach R, Juravski D, Atzil-Slonim D (2020) Using computerized text analysis to examine associations between linguistic features and clients’ distress during psychotherapy. Journal of counseling psychology
2020
Later among the works it cites.
Wolf T, Debut L, Sanh V, Chaumond J, Delangue C, Moi A, Cistac P, Rault T, Louf R, Funtowicz M, Davison J, Shleifer S, von Platen P, Ma C, Jernite Y, Plu J, Xu C, Scao TL, Gugger S, Drame M, Lhoest Q, Rush AM (2020) Transformers: State-of-the-art natural language processing. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , 38–45 (Online: Association for Computational Linguistics), URL https://www.aclweb.org/anthology/2020.emnlp-demos.6
2020
Later among the works it cites.
Yadav A, Vishwakarma DK (2020) Sentiment analysis using deep learning architectures: a review. Artificial Intelligence Review 53(6):4335–4385
2020
Later among the works it cites.
Zhou D, Wu S, Wang Q, Xie J, Tu Z, Li M (2020) Emotion classification by jointly learning to lexiconize and classify. Proceedings of the 28th International Conference on Computational Linguistics , 3235–3245
2020
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…