2020

SBERT-WK: A Sentence Embedding Method by Dissecting BERT-based Word Models

Wang, Bin, Kuo, C. -C. Jay

Understand

Sentence embedding is an important research topic in natural language processing (NLP) since it can transfer knowledge to downstream tasks.

Meanwhile, a contextualized word representation, called BERT, achieves the state-of-the-art performance in quite a few NLP tasks.
Yet, it is an open problem to generate a high quality sentence representation from BERT-based word models.
It was shown in previous study that different layers of BERT capture different linguistic properties.

Built on

X. Li and D. Roth, “Learning question classifiers,” in Proceedings of the 19th international conference on Computational linguistics-Volume 1 . Association for Computational Linguistics, 2002, pp. 1–7
2002
Earlier work this paper cites.
M. Hu and B. Liu, “Mining and summarizing customer reviews,” in Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining , 2004, pp. 168–177
2004
Earlier work this paper cites.
B. Pang and L. Lee, “A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts,” in Proceedings of the 42nd annual meeting on Association for Computational Linguistics . Association for Computational Linguistics, 2004, p. 271
2004
Earlier work this paper cites.
B. Dolan, C. Quirk, and C. Brockett, “Unsupervised construction of large paraphrase corpora: Exploiting massively parallel news sources,” in Proceedings of the 20th international conference on Computational Linguistics . Association for Computational Linguistics, 2004, p. 350
2004
Earlier work this paper cites.
B. Pang and L. Lee, “Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales,” in Proceedings of the 43rd annual meeting on association for computational linguistics . Association for Computational Linguistics, 2005, pp. 115–124
2005
Earlier work this paper cites.
J. Wiebe, T. Wilson, and C. Cardie, “Annotating expressions of opinions and emotions in language,” Language resources and evaluation , vol. 39, no. 2-3, pp. 165–210, 2005
2005
Earlier work this paper cites.
E. Agirre, D. Cer, M. Diab, and A. Gonzalez-Agirre, “Semeval-2012 task 6: A pilot on semantic textual similarity,” in * SEM 2012: The First Joint Conference on Lexical and Computational Semantics–Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012) , 2012, pp. 385–393
2012
Earlier work this paper cites.
T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean, “Distributed representations of words and phrases and their compositionality,” in Advances in neural information processing systems , 2013, pp. 3111–3119
2013
Earlier work this paper cites.
E. Agirre, D. Cer, M. Diab, A. Gonzalez-Agirre, and W. Guo, “* sem 2013 shared task: Semantic textual similarity,” in Second Joint Conference on Lexical and Computational Semantics (* SEM), Volume 1: Proceedings of the Main Conference and the Shared Task: Semantic Textual Similarity , 2013, pp. 32–43
2013
Earlier work this paper cites.
R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Y. Ng, and C. Potts, “Recursive deep models for semantic compositionality over a sentiment treebank,” in Proceedings of the 2013 conference on empirical methods in natural language processing , 2013, pp. 1631–1642
2013
Earlier work this paper cites.
J. Pennington, R. Socher, and C. Manning, “Glove: Global vectors for word representation,” in Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) , 2014, pp. 1532–1543
2014
Earlier work this paper cites.
E. Agirre, C. Banea, C. Cardie, D. Cer, M. Diab, A. Gonzalez-Agirre, W. Guo, R. Mihalcea, G. Rigau, and J. Wiebe, “Semeval-2014 task 10: Multilingual semantic textual similarity,” in Proceedings of the 8th international workshop on semantic evaluation (SemEval 2014) , 2014, pp. 81–91
2014
Earlier work this paper cites.
M. Marelli, S. Menini, M. Baroni, L. Bentivogli, R. Bernardi, R. Zamparelli et al. , “A sick cure for the evaluation of compositional distributional semantic models,” in LREC , 2014, pp. 216–223
2014
Earlier work this paper cites.
R. Kiros, Y. Zhu, R. R. Salakhutdinov, R. Zemel, R. Urtasun, A. Torralba, and S. Fidler, “Skip-thought vectors,” in Advances in neural information processing systems , 2015, pp. 3294–3302
2015
Earlier work this paper cites.
E. Agirre, C. Banea, C. Cardie, D. Cer, M. Diab, A. Gonzalez-Agirre, W. Guo, I. Lopez-Gazpio, M. Maritxalar, R. Mihalcea et al. , “Semeval-2015 task 2: Semantic textual similarity, english, spanish and pilot on interpretability,” in Proceedings of the 9th international workshop on semantic evaluation (SemEval 2015) , 2015, pp. 252–263
2015
Earlier work this paper cites.
E. Agirre, C. Banea, D. Cer, M. Diab, A. Gonzalez-Agirre, R. Mihalcea, G. Rigau, and J. Wiebe, “Semeval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation,” in Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016) , 2016, pp. 497–511
2016
Earlier work this paper cites.
P. Bojanowski, E. Grave, A. Joulin, and T. Mikolov, “Enriching word vectors with subword information,” Transactions of the Association for Computational Linguistics , vol. 5, pp. 135–146, 2017
2017
Earlier work this paper cites.

Similar

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in neural information processing systems , 2017, pp. 5998–6008
2017
Cited alongside, same era.
S. Arora, Y. Liang, and T. Ma, “A simple but tough-to-beat baseline for sentence embeddings,” International Conference on Learning Representations (ICLR) , 2017
2017
Cited alongside, same era.
A. Conneau, D. Kiela, H. Schwenk, L. Barrault, and A. Bordes, “Supervised learning of universal sentence representations from natural language inference data,” in Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing . Copenhagen, Denmark: Association for Computational Linguistics, September 2017, pp. 670–680
2017
Cited alongside, same era.
J. Wieting and K. Gimpel, “Revisiting recurrent networks for paraphrastic sentence embeddings,” Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics , 2017
2017
Cited alongside, same era.
D. Cer, M. Diab, E. Agirre, I. Lopez-Gazpio, and L. Specia, “SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation,” in Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017) . Vancouver, Canada: Association for Computational Linguistics, Aug. 2017, pp. 1–14
2017
Cited alongside, same era.
J. Mu, S. Bhat, and P. Viswanath, “All-but-the-top: Simple and effective postprocessing for word representations,” International Conference on Learning Representations , 2018
2017
Cited alongside, same era.
M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer, “Deep contextualized word representations,” in Proc. of NAACL , 2018
2018
Cited alongside, same era.
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805 , 2018
Original
2018
Cited alongside, same era.
A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever, “Improving language understanding by generative pre-training,” URL https://s3-us-west-2. amazonaws. com/openai-assets/researchcovers/languageunsupervised/language understanding paper. pdf , 2018
2018
Cited alongside, same era.
P. Rajpurkar, R. Jia, and P. Liang, “Know what you don’t know: Unanswerable questions for SQuAD,” in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) . Melbourne, Australia: Association for Computational Linguistics, Jul. 2018, pp. 784–789
2018
Cited alongside, same era.
A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. Bowman, “GLUE: A multi-task benchmark and analysis platform for natural language understanding,” in Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP . Brussels, Belgium: Association for Computational Linguistics, Nov. 2018, pp. 353–355
2018
Cited alongside, same era.
K. Ethayarajh, “Unsupervised random walk sentence embeddings: A strong but simple baseline,” in Proceedings of The Third Workshop on Representation Learning for NLP , 2018, pp. 91–100
2018
Cited alongside, same era.

Then

C. S. Perone, R. Silveira, and T. S. Paula, “Evaluation of sentence embeddings in downstream and linguistic probing tasks,” arXiv preprint arXiv:1806.06259 , 2018
Original
2018
Later among the works it cites.
Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. R. Salakhutdinov, and Q. V. Le, “Xlnet: Generalized autoregressive pretraining for language understanding,” in Advances in neural information processing systems , 2019, pp. 5754–5764
2019
Later among the works it cites.
G. Jawahar, B. Sagot, D. Seddah, S. Unicomb, G. Iñiguez, M. Karsai, Y. Léo, M. Karsai, C. Sarraute, É. Fleury et al. , “What does bert learn about the structure of language?” in 57th Annual Meeting of the Association for Computational Linguistics (ACL), Florence, Italy , 2019
2019
Later among the works it cites.
N. F. Liu, M. Gardner, Y. Belinkov, M. E. Peters, and N. A. Smith, “Linguistic knowledge and transferability of contextual representations,” in Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , 2019
2019
Later among the works it cites.
N. Reimers and I. Gurevych, “Sentence-bert: Sentence embeddings using siamese bert-networks,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing . Association for Computational Linguistics, 11 2019
2019
Later among the works it cites.
Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized bert pretraining approach,” arXiv preprint arXiv:1907.11692 , 2019
Original
2019
Later among the works it cites.
Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut, “Albert: A lite bert for self-supervised learning of language representations,” International Conference on Learning Representations , 2019
2019
Later among the works it cites.
F. Petroni, T. Rocktäschel, P. Lewis, A. Bakhtin, Y. Wu, A. H. Miller, and S. Riedel, “Language models as knowledge bases?” Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing , 2019
2019
Later among the works it cites.
O. Kovaleva, A. Romanov, A. Rogers, and A. Rumshisky, “Revealing the dark secrets of bert,” Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing , 2019
2019
Later among the works it cites.
P. Michel, O. Levy, and G. Neubig, “Are sixteen heads really better than one?” in Advances in Neural Information Processing Systems , 2019, pp. 14 014–14 024
2019
Later among the works it cites.
Y. Hao, L. Dong, F. Wei, and K. Xu, “Visualizing and understanding the effectiveness of bert,” Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing , 2019
2019
Later among the works it cites.
K. Ethayarajh, “How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embeddings,” Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing , 2019
2019
Later among the works it cites.
Z. Yang, C. Zhu, and W. Chen, “Parameter-free sentence embedding via orthogonal basis,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , 2019, pp. 638–648
2019
Later among the works it cites.
K. Ethayarajh, D. Duvenaud, and G. Hirst, “Towards understanding linear word analogies,” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics . Florence, Italy: Association for Computational Linguistics, Jul. 2019, pp. 3253–3262
2019
Later among the works it cites.
S. Eger, A. Rücklé, and I. Gurevych, “Pitfalls in the evaluation of sentence embeddings,” 4th Workshop on Representation Learning for NLP , 2019
2019
Later among the works it cites.
B. Wang, F. Chen, A. Wang, and C.-C. J. Kuo, “Post-processing of word representations via variance normalization and dynamic embedding,” 2019 IEEE International Conference on Multimedia and Expo (ICME) , pp. 718–723, 2019
2019
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…