Revisiting Word Embeddings in the LLM Era

Language models are few-shot learners

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020 · 1901

Earlier work this paper cites.

Roberta: A robustly optimized bert pretraining approach

Original

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019 · 1907

Earlier work this paper cites.

How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embeddings

Original

Kawin Ethayarajh. 2019 · 1909

Earlier work this paper cites.

Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension

Original

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019 · 1910

Earlier work this paper cites.

WordNet: An Electronic Lexical Database

Christiane Fellbaum. 1998 · 1998

Earlier work this paper cites.

Linguistic regularities in continuous space word representations

Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013b · 2013

Earlier work this paper cites.

Linguistic regularities in sparse and explicit word representations

Omer Levy and Yoav Goldberg. 2014 · 2014

Earlier work this paper cites.

GloVe: Global vectors for word representation

Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014 · 2014

Earlier work this paper cites.

Word embeddings, analogies, and machine learning: Beyond king - man + woman = queen

Aleksandr Drozd, Anna Gladkova, and Satoshi Matsuoka. 2016 · 2016

Earlier work this paper cites.

Analogy-based detection of morphological and semantic relations with word embeddings: what works and what doesn’t

Anna Gladkova, Aleksandr Drozd, and Satoshi Matsuoka. 2016 · 2016

Earlier work this paper cites.

Issues in evaluating semantic spaces using word analogies

Original

Tal Linzen. 2016 · 2016

Earlier work this paper cites.

Universal sentence encoder

Original

Daniel Cer, Yinfei Yang, Sheng yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2018 · 2018

Earlier work this paper cites.

Bert: Pre-training of deep bidirectional transformers for language understanding

Original

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018 · 2018

Earlier work this paper cites.

Dissecting contextual word embeddings: Architecture and representation

Original

Matthew E Peters, Mark Neumann, Luke Zettlemoyer, and Wen-tau Yih. 2018 · 2018

Cited alongside, same era.

Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond

Mikel Artetxe and Holger Schwenk. 2019 · 2019

Cited alongside, same era.

Sentence-bert: Sentence embeddings using siamese bert-networks

Original

Nils Reimers and Iryna Gurevych. 2019 · 2019

Cited alongside, same era.

Analogies minus analogy test: measuring regularities in word embeddings

Original

Louis Fournier, Emmanuel Dupoux, and Ewan Dunbar. 2020 · 2020

Cited alongside, same era.

Contextual and non-contextual word embeddings: an in-depth linguistic investigation

Alessio Miaschi and Felice Dell’Orletta. 2020 · 2020

Palm 2 technical report

Original

Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. 2023 · 2023

Later among the works it cites.

Sparks of artificial general intelligence: Early experiments with gpt-4

Original

Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023 · 2023

Later among the works it cites.

The daunting dilemma with sentence encoders: Success on standard benchmarks, failure in capturing basic semantic properties

Original

Yash Mahajan, Naman Bansal, and Shubhra Kanti Karmaker. 2023 · 2023

Later among the works it cites.

Zero-shot multi-label topic inference with sentence encoders and llms

Souvika Sarkar, Dongji Feng, and Shubhra Kanti Karmaker Santu. 2023 · 2023

Later among the works it cites.