A Survey on Transformer Context Extension: Approaches and Evaluation

Bertscore: Evaluating text generation with bert. arxiv

Original

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2020 · 1904

Earlier work this paper cites.

Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model

Original

Alexander R Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir R Radev. 2019 · 1906

Earlier work this paper cites.

Bigpatent: A large-scale dataset for abstractive and coherent summarization

Original

Eva Sharma, Chen Li, and Lu Wang. 2019 · 1906

Earlier work this paper cites.

Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension

Original

M Lewis. 2019 · 1910

Earlier work this paper cites.

Learning question classifiers

Xin Li and Dan Roth. 2002 · 2002

Earlier work this paper cites.

Bleu: a method for automatic evaluation of machine translation

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002 · 2002

Earlier work this paper cites.

Longformer: The long-document transformer

Original

Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020 · 2004

Earlier work this paper cites.

Rouge: A package for automatic evaluation of summaries

Chin-Yew Lin. 2004 · 2004

Earlier work this paper cites.

Dialogue-based relation extraction

Original

Dian Yu, Kai Sun, Claire Cardie, and Dong Yu. 2020 · 2004

Earlier work this paper cites.

Language models are few-shot learners

Original

Tom B Brown. 2020 · 2005

Earlier work this paper cites.

Evaluating evaluation methods for generation in the presence of variation

Amanda Stent, Matthew Marge, and Mohit Singhai. 2005 · 2005

Earlier work this paper cites.

An investigation into the validity of some metrics for automatically evaluating natural language generation systems

Ehud Reiter and Anja Belz. 2009 · 2009

Earlier work this paper cites.

Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps

Original

Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020 · 2011

Earlier work this paper cites.

Task definition for large scale text categorization at nlpcc 2014

NLPCC. 2014 · 2014

Earlier work this paper cites.

Squad: 100,000+ questions for machine comprehension of text

Original

P Rajpurkar. 2016 · 2016

Earlier work this paper cites.

Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension

Original

Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017 · 2017

Earlier work this paper cites.

Attention is all you need

A Vaswani. 2017 · 2017

Earlier work this paper cites.

Bert: Pre-training of deep bidirectional transformers for language understanding

Original

Jacob Devlin. 2018 · 2018

Earlier work this paper cites.

The narrativeqa reading comprehension challenge

Tomáš Kočiskỳ, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2018 · 2018

Earlier work this paper cites.

Improving language understanding by generative pre-training

Alec Radford. 2018 · 2018

Earlier work this paper cites.

Hotpotqa: A dataset for diverse, explainable multi-hop question answering

Original

Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018 · 2018

Earlier work this paper cites.

Transformer-xl: Attentive language models beyond a fixed-length context

Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019 · 2019

Earlier work this paper cites.

Exploring the limits of transfer learning with a unified text-to-text transformer

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020 · 2020

Earlier work this paper cites.

Fourier features let networks learn high frequency functions in low dimensional domains

Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. 2020 · 2020

Earlier work this paper cites.

Extractive opinion summarization in quantized transformer spaces

Stefanos Angelidis, Reinald Kim Amplayo, Yoshihiko Suhara, Xiaolan Wang, and Mirella Lapata. 2021 · 2021

Earlier work this paper cites.

Training verifiers to solve math word problems

Original

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021 · 2021

Earlier work this paper cites.

Efficient attentions for long document summarization

Original

Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. 2021 · 2021

Earlier work this paper cites.

Bidimensional leaderboards: Generate and evaluate language hand in hand

Original

Jungo Kasai, Keisuke Sakaguchi, Ronan Le Bras, Lavinia Dunagan, Jacob Morrison, Alexander R Fabbri, Yejin Choi, and Noah A Smith. 2021 · 2021

Earlier work this paper cites.

Booksum: A collection of datasets for long-form narrative summarization

Original

Wojciech Kryściński, Nazneen Rajani, Divyansh Agarwal, Caiming Xiong, and Dragomir Radev. 2021 · 2021

Earlier work this paper cites.

Train short, test long: Attention with linear biases enables input length extrapolation

Original

Ofir Press, Noah A Smith, and Mike Lewis. 2021 · 2021

Earlier work this paper cites.

Efficient content-based sparse attention with routing transformers

Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. 2021 · 2021

Earlier work this paper cites.

Do long-range language models actually use long-range context?

Original

Simeng Sun, Kalpesh Krishna, Andrew Mattarella-Micke, and Mohit Iyyer. 2021 · 2021

Earlier work this paper cites.

GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model

Ben Wang and Aran Komatsuzaki. 2021 · 2021

Earlier work this paper cites.

Bartscore: Evaluating generated text as text generation

Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021 · 2021

Earlier work this paper cites.

Qmsum: A new benchmark for query-based multi-domain meeting summarization

Original

Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, et al. 2021 · 2021

Earlier work this paper cites.

Recurrent memory transformer

Aydar Bulatov, Yury Kuratov, and Mikhail Burtsev. 2022 · 2022

Earlier work this paper cites.

Dialogen: Generalized long-range context representation for dialogue systems

Original

Suvodip Dey, Maunendra Sankar Desarkar, Asif Ekbal, and PK Srijith. 2022 · 2022

Cited alongside, same era.

Structured prompting: Scaling in-context learning to 1,000 examples

Original

Yaru Hao, Yutao Sun, Li Dong, Zhixiong Han, Yuxian Gu, and Furu Wei. 2022 · 2022

Cited alongside, same era.

Parallel context windows for large language models

Original

Nir Ratner, Yoav Levine, Yonatan Belinkov, Ori Ram, Inbal Magar, Omri Abend, Ehud Karpas, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2022 · 2022

Cited alongside, same era.

Scrolls: Standardized comparison over long language sequences

Original

Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, et al. 2022 · 2022

Cited alongside, same era.

Efficient streaming language models with attention sinks

Original

Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2023 · 2023

Later among the works it cites.

Megabyte: Predicting million-byte sequences with multiscale transformers

Lili Yu, Dániel Simig, Colin Flaherty, Armen Aghajanyan, Luke Zettlemoyer, and Mike Lewis. 2023 · 2023

Later among the works it cites.

Length extrapolation of transformers: A survey from the perspective of position encoding

Original

Liang Zhao, Xiaocheng Feng, Xiachong Feng, Bin Qin, and Ting Liu. 2023 · 2023

Later among the works it cites.

Codegeex: A pre-trained model for code generation with multilingual benchmarking on humaneval-x

Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Lei Shen, Zihan Wang, Andi Wang, Yang Li, et al. 2023 · 2023

Later among the works it cites.

Through the lens of core competency: Survey on evaluation of large language models