Fetching the paper…

Reading the bibliography…

2022

Global memory transformer for processing long documents

Adel, Arij Al

Original
PDF
Post on X

Understand

Transformer variants dominate the state-of-the-art in different natural language processing tasks such as translation, reading comprehension and summarization.

Our paper is more directed to use general memory slots added to the inputs and studying the results of adding these slots.
This paper is a go on study of general memory slots rule that were added to the input of the proposed model in previous work.
We have two main tasks;1) pretraining task using masked language modeling and b) fine tuning task using HotpotQA .

Global memory transformer for processing long documents · Around

Built on

Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W.W., Salakhutdinov, R., Manning, C.D.: Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In: EMNLP (2018)
2018
Earlier work this paper cites.
Mihaylova, T., Martins, A.F.T.: Scheduled sampling for transformers. CoRR abs/1906.07651 (2019)
Original
2019
Earlier work this paper cites.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P.J.: Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683 (2019)
Original
2019
Earlier work this paper cites.
Ainslie, J., Ontañón, S., Alberti, C., Pham, P., Ravula, A., Sanghai, S.K.: Etc: Encoding long and structured data in transformers. ArXiv abs/2004.08483 (2020)
Original
2020
Earlier work this paper cites.

Similar

Beltagy, I., Peters, M.E., Cohan, A.: Longformer: The long-document transformer. ArXiv abs/2004.05150 (2020)
Original
2020
Cited alongside, same era.
Geng, X., Wang, L., Wang, X., Qin, B., Liu, T., Tu, Z.: How does selective mechanism improve self-attention networks? CoRR abs/2005.00979 (2020)
Original
2020
Cited alongside, same era.
Gupta, A., Berant, J.: Gmat: Global memory augmentation for transformers. ArXiv abs/2006.03274 (2020)
Original
2020
Cited alongside, same era.

Then

Adel, A.A., Burtsev, M.S.: Memory transformer with hierarchical attention for long document processing. 2021 International Conference Engineering and Telecommunication (En&T) pp. 1–7 (2021)
2021
Later among the works it cites.
Guo, M., Ainslie, J., Uthus, D.C., Ontañón, S., Ni, J., Sung, Y.H., Yang, Y.: Longt5: Efficient text-to-text transformer for long sequences. ArXiv abs/2112.07916 (2021)
Original
2021
Later among the works it cites.
Jaszczur, S., Chowdhery, A., Mohiuddin, A., Kaiser, L., Gajewski, W., Michalewski, H., Kanerva, J.: Sparse is enough in scaling transformers. CoRR abs/2111.12763 (2021)
Original
2021
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…