A Text is Worth Several Tokens: Text Embedding from LLMs Secretly Aligns Well with The Key Tokens

Language models are few-shot learners

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020 · 1901

Earlier work this paper cites.

Roberta: A robustly optimized bert pretraining approach

Original

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019 · 1907

Earlier work this paper cites.

Www’18 open challenge: financial opinion mining and question answering

Macedo Maia, Siegfried Handschuh, André Freitas, Brian Davis, Ross McDermott, Manel Zarrouk, and Alexandra Balahur. 2018 · 1942

Earlier work this paper cites.

Nltk: The natural language toolkit

Edward Loper and Steven Bird. 2002 · 2002

Earlier work this paper cites.

A large annotated corpus for learning natural language inference

Samuel Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015 · 2015

Earlier work this paper cites.

A full-text learning to rank dataset for medical information retrieval

Vera Boteva, Demian Gholipour, Artem Sokolov, and Stefan Riezler. 2016 · 2016

Earlier work this paper cites.

Ms marco: A human-generated machine reading comprehension dataset

Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016 · 2016

Earlier work this paper cites.

Improving language understanding by generative pre-training

Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018 · 2018

Earlier work this paper cites.

Carer: Contextualized affect representations for emotion recognition

Elvis Saravia, Hsien-Chi Toby Liu, Yen-Hao Huang, Junlin Wu, and Yi-Shin Chen. 2018 · 2018

Earlier work this paper cites.

Retrieval of the best counterargument without prior topic knowledge

Henning Wachsmuth, Shahbaz Syed, and Benno Stein. 2018 · 2018

Earlier work this paper cites.

Bert: Pre-training of deep bidirectional transformers for language understanding

Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019 · 2019

Earlier work this paper cites.

Privacy risks of general-purpose language models

Xudong Pan, Mi Zhang, Shouling Ji, and Min Yang. 2020 · 2020

Earlier work this paper cites.

Fact or fiction: Verifying scientific claims

David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020 · 2020

Cited alongside, same era.

A mathematical framework for transformer circuits

Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. 2021 · 2021

Cited alongside, same era.

Simcse: Simple contrastive learning of sentence embeddings

Original

Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021 · 2021

Cited alongside, same era.

Lora: Low-rank adaptation of large language models

Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2021 · 2021

Cited alongside, same era.

Analyzing transformers in embedding space

Original

Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. 2022 · 2022

Cited alongside, same era.

Text embeddings reveal (almost) as much as text

John Morris, Volodymyr Kuleshov, Vitaly Shmatikov, and Alexander M Rush. 2023 · 2023

Later among the works it cites.

What are you token about? dense retrieval as distributions over the vocabulary

Ori Ram, Liat Bezalel, Adi Zicher, Yonatan Belinkov, Jonathan Berant, and Amir Globerson. 2023 · 2023

Later among the works it cites.

One embedder, any task: Instruction-finetuned text embeddings

Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A Smith, Luke Zettlemoyer, and Tao Yu. 2023 · 2023

Later among the works it cites.

Improving text embeddings with large language models

Original

Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2023 · 2023

Later among the works it cites.

Llm2vec: Large language models are secretly powerful text encoders

Original

Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. 2024 · 2024