Semformer: Transformer Language Models with Semantic Planning

Augmenting self-attention with persistent memory

Original

Sainbayar Sukhbaatar, Edouard Grave, Guillaume Lample, Hervé Jégou, and Armand Joulin. 2019 · 1907

Earlier work this paper cites.

A learning algorithm for continually running fully recurrent neural networks

Ronald J. Williams and David Zipser. 1989 · 1989

Earlier work this paper cites.

ROUGE: A package for automatic evaluation of summaries

Chin-Yew Lin. 2004 · 2004

Earlier work this paper cites.

Automatically constructing a corpus of sentential paraphrases

William B. Dolan and Chris Brockett. 2005 · 2005

Earlier work this paper cites.

Memory transformer

Original

Mikhail S. Burtsev and Grigory V. Sapunov. 2020 · 2006

Earlier work this paper cites.

Lower bounds for reductions

Matti Kääriäinen. 2006 · 2006

Earlier work this paper cites.

Neural net language models

Yoshua Bengio. 2008 · 2008

Earlier work this paper cites.

Efficient reductions for imitation learning

Stephane Ross and Drew Bagnell. 2010 · 2010

Earlier work this paper cites.

Recursive deep models for semantic compositionality over a sentiment treebank

Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013 · 2013

Earlier work this paper cites.

Auto-encoding variational bayes

Diederik P. Kingma and Max Welling. 2014 · 2014

Earlier work this paper cites.

Pointer sentinel mixture models

Original

Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016 · 2016

Earlier work this paper cites.

The LAMBADA dataset: Word prediction requiring a broad discourse context

Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. 2016 · 2016

Earlier work this paper cites.

Attention is all you need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, undefinedukasz Kaiser, and Illia Polosukhin. 2017 · 2017

Earlier work this paper cites.

Non-autoregressive neural machine translation

Jiatao Gu, James Bradbury, Caiming Xiong, Victor O. K. Li, and Richard Socher. 2018 · 2018

Earlier work this paper cites.

Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization

Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018 · 2018

Earlier work this paper cites.

SAMSum corpus: A human-annotated dialogue dataset for abstractive summarization

Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. 2019 · 2019

Earlier work this paper cites.

Language models are unsupervised multitask learners

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019 · 2019

Earlier work this paper cites.

Language models are few-shot learners

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020 · 2020

Earlier work this paper cites.

Momentum contrast for unsupervised visual representation learning

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020 · 2020

Cited alongside, same era.

Optimus: Organizing sentences via pre-trained modeling of a latent space

Chunyuan Li, Xiang Gao, Yuan Li, Baolin Peng, Xiujun Li, Yizhe Zhang, and Jianfeng Gao. 2020 · 2020

Cited alongside, same era.

Prophetnet: Predicting future n-gram for sequence-to-sequence pre-training

Weizhen Qi, Yu Yan, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang, and Ming Zhou. 2020 · 2020

Cited alongside, same era.

Dialogsum challenge: Summarizing real-life scenario dialogues

Yulong Chen, Yang Liu, and Yue Zhang. 2021 · 2021

Cited alongside, same era.

The power of scale for parameter-efficient prompt tuning

Brian Lester, Rami Al-Rfou, and Noah Constant. 2021 · 2021

Cited alongside, same era.

Prefix-tuning: Optimizing continuous prompts for generation

Embers of autoregression: Understanding large language models through the problem they are trained to solve

Original

R. Thomas McCoy, Shunyu Yao, Dan Friedman, Matthew Hardy, and Thomas L. Griffiths. 2023 · 2023

Later among the works it cites.

Pass: Parallel speculative sampling

Original

Giovanni Monea, Armand Joulin, and Edouard Grave. 2023 · 2023

Later among the works it cites.

Learning to compress prompts with gist tokens

Jesse Mu, Xiang Li, and Noah D. Goodman. 2023 · 2023

Later among the works it cites.

GPT-4 technical report

Original

OpenAI. 2023 · 2023

Later among the works it cites.

Are emergent abilities of large language models a mirage?

Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. 2023 · 2023

Later among the works it cites.