Fetching the paper…
Reading the bibliography…
How does word frequency in pre-training data affect the behavior of similarity metrics in contextualized BERT embeddings? Are there systematic ways in which some word relationships are exaggerated or understated? In this work, we explore the geometric characteristics of contextualized word embeddings with two novel tools: (1) an identity probe that predicts the identity of a word using its embedding; (2) the minimal bounding sphere for a word's contextualized representations.
Sentence-bert: Sentence embeddings using siamese bert-networks
Nils Reimers and Iryna Gurevych. 2019 · 1908
Earlier work this paper cites.
The meaning-frequency relationship of words
George Kingsley Zipf. 1945 · 1945
Earlier work this paper cites.
Symbol grounding and meaning: A comparison of high-dimensional and embodied theories of meaning
Arthur M. Glenberg and David A. Robertson. 2000 · 2000
Earlier work this paper cites.
Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr Nisnevich, Nicolas Pinto, and Joseph Turian. 2020 · 2004
Earlier work this paper cites.
Word meaning in minds and machines
Brenden M Lake and Gregory L Murphy. 2020 · 2008
Earlier work this paper cites.
Improving word representations via global context and multiple word prototypes
Eric H Huang, Richard Socher, Christopher D Manning, and Andrew Y Ng. 2012 · 2012
Earlier work this paper cites.
Neural word embedding as implicit matrix factorization
Omer Levy and Yoav Goldberg. 2014 · 2014
Earlier work this paper cites.
Aligning books and movies: Towards story-like visual explanations by watching movies and reading books
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015 · 2015
Cited alongside, same era.
Bad company—neighborhoods in neural embedding spaces considered harmful
Johannes Hellrich and Udo Hahn. 2016 · 2016
Cited alongside, same era.
The strange geometry of skip-gram with negative sampling
David Mimno and Laure Thompson. 2017 · 2017
Cited alongside, same era.
Data statements for natural language processing: Toward mitigating system bias and enabling better science
Emily M Bender and Batya Friedman. 2018 · 2018
Cited alongside, same era.
Bert: Pre-training of deep bidirectional transformers for language understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018 · 2018
Cited alongside, same era.
How western journalists actually write about africa
Toussaint Nothias. 2018 · 2018
Later among the works it cites.
How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embeddings
Kawin Ethayarajh. 2019 · 2019
Later among the works it cites.
Model cards for model reporting
Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019 · 2019
Later among the works it cites.
Visualizing and measuring the geometry of bert
Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda B Viegas, Andy Coenen, Adam Pearce, and Been Kim. 2019 · 2019
Later among the works it cites.
Climbing towards nlu: On meaning, form, and understanding in the age of data
Emily M Bender and Alexander Koller. 2020 · 2020
Later among the works it cites.
Factors influencing the surprising instability of word embeddings
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. 2018 · 2018
Cited alongside, same era.
NILC at CWI 2018: Exploring feature engineering and feature learning
Nathan Hartmann and Leandro Borges dos Santos. 2018 · 2018
Cited alongside, same era.
Towards understanding linear word analogies
Kawin Ethayarajh, David Duvenaud, and Graeme Hirst. 2019a
Cited in the paper.
Understanding undesirable word embedding associations
Kawin Ethayarajh, David Duvenaud, and Graeme Hirst. 2019b
Cited in the paper.
Laura Wendlandt, Jonathan K. Kummerfeld, and Rada Mihalcea. 2018 · 2092
Closest in time.