Logarithmic Memory Networks (LMNs): Efficient Long-Range Sequence Modeling for Resource-Constrained Environments

J. L. Elman, “Finding structure in time,” Cognitive Science , vol. 14, no. 2, pp. 179–211, 1990

1990

Earlier work this paper cites.

S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural computation , vol. 9, no. 8, pp. 1735–1780, 1997

1997

Earlier work this paper cites.

T. Mikolov, M. Karafiát, L. Burget, J. Cernockỳ, and S. Khudanpur, “Recurrent neural network based language model,” Interspeech , vol. 2, pp. 1045–1048, 2010

2010

Earlier work this paper cites.

K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio, “Learning phrase representations using rnn encoder-decoder for statistical machine translation,” arXiv preprint arXiv:1406.1078 , 2014

Original

2014

Earlier work this paper cites.

K. Cho and Y. Bengio, “Properties of the lstm and gru networks,” arXiv preprint arXiv:1412.3555 , 2014

Original

2014

Earlier work this paper cites.

A. Graves, G. Wayne, and I. Danihelka, “Neural turing machines,” arXiv preprint arXiv:1410.5401 , 2014

Original

2014

Earlier work this paper cites.

J. Weston, S. Chopra, and A. Bordes, “Memory networks,” arXiv preprint arXiv:1410.3916 , 2014

Original

2014

Earlier work this paper cites.

D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” International Conference on Learning Representations (ICLR) , 2015

2015

Earlier work this paper cites.

A. Graves, G. Wayne, M. Reynolds, T. Harley, I. Danihelka, A. Grabska-Barwińska, S. G. Colmenarejo, J. Światkowski, D. Tan, S. Mohamed et al. , “Hybrid computing using a neural network with dynamic external memory,” Nature , vol. 538, no. 7626, pp. 471–476, 2016

2016

Earlier work this paper cites.

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems , vol. 30, 2017

2017

Earlier work this paper cites.

N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean, “Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,” arXiv preprint arXiv:1701.06538 , 2017. [Online]. Available: https://arxiv.org/abs/1701.06538

Original

2017

Earlier work this paper cites.

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805 , 2018

Original

2018

Cited alongside, same era.

A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever, “Improving language understanding by generative pre-training,” OpenAI Blog , 2018

2018

Cited alongside, same era.

Z. Dai, Z. Yang, Y. Yang, J. Carbonell, Q. V. Le, and R. Salakhutdinov, “Transformer-xl: Attentive language models beyond a fixed-length context,” arXiv preprint arXiv:1901.02860 , 2019

Original

2019

Cited alongside, same era.

M. E. Beltagy Iz Peters and C. Arman, “Longformer: The long-document transformer,” arXiv preprint arXiv:2004.05150 , 2020

Original

2020

Cited alongside, same era.

S. Wang, N. Liu, W. Han, M. Yu, Y. Wang, Y. Yang, and K. Cho, “Linformer: Self-attention with linear complexity,” arXiv preprint arXiv:2006.04768 , 2020

M. Tan and S. Liu, “Hybrid memory networks for sequential data processing,” ICML , 2022

2022

Later among the works it cites.

Y. Tay, M. Dehghani et al. , “Synthformer: Beyond token-level self-attention,” NeurIPS , 2022

2022

Later among the works it cites.

A. Gu and T. Dao, “Scaling structured state space models for long sequences,” NeurIPS , 2022

2022

Later among the works it cites.

W. Fedus, B. Zoph, and N. Shazeer, “Switch transformers: Scaling models with mixture-of-experts layers,” Nature Machine Intelligence , 2022

2022

Later among the works it cites.

R. Dutta et al. , “Neural memory architectures for sequential decision-making,” ICLR , 2023

2023

Later among the works it cites.

H. Liu and H. Rao, “Recurrent attention mechanisms for efficient long-sequence processing,” ICML , 2023