Venturing into Uncharted Waters: The Navigation Compass from Transformer to Mamba

A new approach to linear filtering and prediction problems

Rudolph Emil Kalman · 1960

Earlier work this paper cites.

A model for reasoning about persistence and causation

Thomas Dean and Keiji Kanazawa · 1989

Earlier work this paper cites.

Modern control theory

William L Brogan · 1991

Earlier work this paper cites.

Probabilistic graphical models: principles and techniques

Daphne Koller and Nir Friedman · 2009

Earlier work this paper cites.

Time series analysis by state space methods , volume 38

James Durbin and Siem Jan Koopman · 2012

Earlier work this paper cites.

U-net: Convolutional networks for biomedical image segmentation

Olaf Ronneberger, Philipp Fischer, and Thomas Brox · 2015

Earlier work this paper cites.

ImageNet Large Scale Visual Recognition Challenge

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei · 2015

Earlier work this paper cites.

Attention is all you need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin · 2017

Earlier work this paper cites.

Bert: Pre-training of deep bidirectional transformers for language understanding

Original

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova · 2018

Earlier work this paper cites.

Improving language understanding by generative pre-training

Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al · 2018

Earlier work this paper cites.

Transformer dissection: a unified understanding of transformer’s attention via the lens of kernel

Original

Yao-Hung Hubert Tsai, Shaojie Bai, Makoto Yamada, Louis-Philippe Morency, and Ruslan Salakhutdinov · 2019

Earlier work this paper cites.

Graph transformer networks

Seongjun Yun, Minbyul Jeong, Raehyun Kim, Jaewoo Kang, and Hyunwoo J Kim · 2019

Earlier work this paper cites.

An image is worth 16x16 words: Transformers for image recognition at scale

Original

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al · 2020

Earlier work this paper cites.

Message passing neural networks

Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl · 2020

Earlier work this paper cites.

Hippo: Recurrent memory with optimal polynomial projections

Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré · 2020

Earlier work this paper cites.

Framing rnn as a kernel method: A neural ode approach

Adeline Fermanian, Pierre Marion, Jean-Philippe Vert, and Gérard Biau · 2021

Earlier work this paper cites.

Swin transformer: Hierarchical vision transformer using shifted windows

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo · 2021

Earlier work this paper cites.

Flashattention: Fast and memory-efficient exact attention with io-awareness

Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré · 2022

Earlier work this paper cites.

Long range graph benchmark

Vijay Prakash Dwivedi, Ladislav Rampášek, Michael Galkin, Ali Parviz, Guy Wolf, Anh Tuan Luu, and Dominique Beaini · 2022

Earlier work this paper cites.

Hungry hungry hippos: Towards language modeling with state space models

Original

Daniel Y Fu, Tri Dao, Khaled K Saab, Armin W Thomas, Atri Rudra, and Christopher Ré · 2022

Earlier work this paper cites.

How to train your hippo: State space models with generalized orthogonal basis projections

Original

Albert Gu, Isys Johnson, Aman Timalsina, Atri Rudra, and Christopher Ré · 2022

Earlier work this paper cites.

Recipe for a general, powerful, scalable graph transformer

Ladislav Rampášek, Michael Galkin, Vijay Prakash Dwivedi, Anh Tuan Luu, Guy Wolf, and Dominique Beaini · 2022

Earlier work this paper cites.

Pretraining without attention

Original

Junxiong Wang, Jing Nathan Yan, Albert Gu, and Alexander M Rush · 2022

Earlier work this paper cites.

Efficient long sequence modeling via state space augmented transformer

Original

Simiao Zuo, Xiaodong Liu, Jian Jiao, Denis Charles, Eren Manavoglu, Tuo Zhao, and Jianfeng Gao · 2022

Earlier work this paper cites.

Flashattention-2: Faster attention with better parallelism and work partitioning

Original

Tri Dao · 2023

Earlier work this paper cites.

On over-squashing in message passing neural networks: The impact of width, depth, and topology

Francesco Di Giovanni, Lorenzo Giusti, Federico Barbero, Giulia Luise, Pietro Lio, and Michael M Bronstein · 2023

Earlier work this paper cites.

Block-state transformer

Original

Mahan Fathi, Jonathan Pilault, Pierre-Luc Bacon, Christopher Pal, Orhan Firat, and Ross Goroshin · 2023

Earlier work this paper cites.

Simple hardware-efficient long convolutions for sequence modeling

Daniel Y Fu, Elliot L Epstein, Eric Nguyen, Armin W Thomas, Michael Zhang, Tri Dao, Atri Rudra, and Christopher Ré · 2023

Earlier work this paper cites.

Mamba: Linear-time sequence modeling with selective state spaces

Original

Albert Gu and Tri Dao · 2023

Earlier work this paper cites.

Hyena hierarchy: Towards larger convolutional language models

Original

Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré · 2023

Earlier work this paper cites.

A survey on oversmoothing in graph neural networks

Original

T Konstantin Rusch, Michael M Bronstein, and Siddhartha Mishra · 2023

Earlier work this paper cites.

Large language models can be easily distracted by irrelevant context

Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H Chi, Nathanael Schärli, and Denny Zhou · 2023

Earlier work this paper cites.

Blackmamba: Mixture of experts for state-space models

Original

Quentin Anthony, Yury Tokpanov, Paolo Glorioso, and Beren Millidge · 2024

Earlier work this paper cites.

Vim-unet: Vision mamba for biomedical segmentation, 2024

Anwai Archit and Constantin Pape · 2024

Cited alongside, same era.

Simple linear attention language models balance the recall-throughput tradeoff

Original

Simran Arora, Sabri Eyuboglu, Michael Zhang, Aman Timalsina, Silas Alberti, Dylan Zinsley, James Zou, Atri Rudra, and Christopher Ré · 2024

Cited alongside, same era.

Retinexmamba: Retinex-based mamba for low-light image enhancement, 2024

Jiesong Bai, Yuhao Yin, Qiyuan He, Yuanxian Li, and Xiaofeng Zhang · 2024

Cited alongside, same era.

Graph mamba: Towards learning on graphs with state space models

Original

Ali Behrouz and Farnoosh Hashemi · 2024

Cited alongside, same era.

Locost: State-space models for long document abstractive summarization

Original

Florian Le Bronnec, Song Duong, Mathieu Ravaut, Alexandre Allauzen, Nancy F Chen, Vincent Guigue, Alberto Lumbreras, Laure Soulier, and Patrick Gallinari · 2024

Efficientvmamba: Atrous selective scan for light weight visual mamba

Original

Xiaohuan Pei, Tao Huang, and Chang Xu · 2024

Closest in time.

Fusionmamba: Efficient image fusion with state space model

Original

Siran Peng, Xiangyu Zhu, Haoyu Deng, Zhen Lei, and Liang-Jian Deng · 2024

Closest in time.

Moe-mamba: Efficient selective state space models with mixture of experts

Original

Maciej Pióro, Kamil Ciebiera, Krystian Król, Jan Ludziejewski, and Sebastian Jaszczur · 2024

Closest in time.

Smcd: High realism motion style transfer via mamba-based diffusion, 2024

Ziyun Qian, Zeyu Xiao, Zhenyi Wu, Dingkang Yang, Mingcheng Li, Shunli Wang, Shuaibing Wang, Dongliang Kou, and Lihua Zhang · 2024

Closest in time.

Vl-mamba: Exploring state space models for multimodal learning

Original