Taking a Deep Breath: Enhancing Language Modeling of Large Language Models with Sentinel Tokens

Transformer-xl: Attentive language models beyond a fixed-length context

Original

Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. 2019 · 1901

Earlier work this paper cites.

Generating long sequences with sparse transformers

Original

Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019 · 1904

Earlier work this paper cites.

Mlqa: Evaluating cross-lingual extractive question answering

Original

Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2019 · 1910

Earlier work this paper cites.

Scaling laws for neural language models

Original

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020 · 2001

Earlier work this paper cites.

Longformer: The long-document transformer

Original

Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020 · 2004

Earlier work this paper cites.

Rouge: A package for automatic evaluation of summaries

Chin-Yew Lin. 2004 · 2004

Earlier work this paper cites.

Pointer sentinel mixture models

Original

Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016 · 2016

Earlier work this paper cites.

Bert: Pre-training of deep bidirectional transformers for language understanding

Original

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018 · 2018

Earlier work this paper cites.

Fixing weight decay regularization in adam

Ilya Loshchilov and Frank Hutter. 2018 · 2018

Earlier work this paper cites.

Multi-news: a large-scale multi-document summarization dataset and abstractive hierarchical model

Original

Alexander R. Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir R. Radev. 2019 · 2019

Earlier work this paper cites.

Opt: Open pre-trained transformer language models

Original

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022 · 2019

Cited alongside, same era.

Transformers: State-of-the-art natural language processing

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2020 · 2020

Cited alongside, same era.

Big bird: Transformers for longer sequences

Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020 · 2020

Cited alongside, same era.

A general language assistant as a laboratory for alignment

Original

Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. 2021 · 2021

Cited alongside, same era.

Lora: Low-rank adaptation of large language models

Adapting language models to compress contexts

Original

Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. 2023 · 2023

Later among the works it cites.

Redpajama: an open dataset for training large language models

Together Computer. 2023 · 2023

Later among the works it cites.

In-context autoencoder for context compression in a large language model

Original

Tao Ge, Jing Hu, Xun Wang, Si-Qing Chen, and Furu Wei. 2023 · 2023

Later among the works it cites.

Mistral 7b

Original

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023 · 2023

Later among the works it cites.

Learning to compress prompts with gist tokens

Original