Fetching the paper…
Reading the bibliography…
Mamba state-space models (SSMs) have recently outperformed state-of-the-art (SOTA) Transformer large language models (LLMs) in various tasks and been widely adapted.
Learning long-term dependencies with gradient descent is difficult
Yoshua Bengio, Patrice Simard, and Paolo Frasconi · 1994
Earlier work this paper cites.
Real-time computation at the edge of chaos in recurrent neural networks
Nils Bertschinger and Thomas Natschläger · 2004
Earlier work this paper cites.
At the edge of chaos: Real-time computations and self-organized criticality in recurrent neural networks
Nils Bertschinger, Thomas Natschläger, and Robert Legenstein · 2004
Earlier work this paper cites.
Large deviations of lyapunov exponents
Tanguy Laffargue, Khanh-Dang Nguyen Thu Lam, Jorge Kurchan, and Julien Tailleur · 2013
Earlier work this paper cites.
On the difficulty of training recurrent neural networks
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio · 2013
Earlier work this paper cites.
Introduction to the modeling and analysis of complex systems
Hiroki Sayama · 2015
Earlier work this paper cites.
A recurrent neural network without chaos
Thomas Laurent and James von Brecht · 2017
Earlier work this paper cites.
Mixed precision training
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al · 2018
Earlier work this paper cites.
Parameter-efficient transfer learning for nlp
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly · 2019
Earlier work this paper cites.
A study of bfloat16 for deep learning training
Dhiraj Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi, Dipankar Das, Kunal Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, Nataraj Jammalamadaka, Jianyu Huang, Hector Yuen, et al · 2019
Earlier work this paper cites.
Language models are few-shot learners
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al · 2020
Earlier work this paper cites.
Measuring massive multitask language understanding
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt · 2020
Earlier work this paper cites.
Beyond exploding and vanishing gradients: analysing rnn training using attractors and smoothness
Antônio H Ribeiro, Koen Tiels, Luis A Aguirre, and Thomas Schön · 2020
Earlier work this paper cites.
Integer quantization for deep learning inference: Principles and empirical evaluation
Hao Wu, Patrick Judd, Xiaojie Zhang, Mikhail Isaev, and Paulius Micikevicius · 2020
Earlier work this paper cites.
Towards a unified view of parameter-efficient transfer learning
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig · 2021
Earlier work this paper cites.
Lora: Low-rank adaptation of large language models
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen · 2021
Earlier work this paper cites.
The power of scale for parameter-efficient prompt tuning
Brian Lester, Rami Al-Rfou, and Noah Constant · 2021
Earlier work this paper cites.
Prefix-tuning: Optimizing continuous prompts for generation
Xiang Lisa Li and Percy Liang · 2021
Earlier work this paper cites.
Winogrande: An adversarial winograd schema challenge at scale
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi · 2021
Earlier work this paper cites.
An explanation of in-context learning as implicit bayesian inference
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma · 2021
Cited alongside, same era.
Flashattention: Fast and memory-efficient exact attention with io-awareness
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré · 2022
Cited alongside, same era.
Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer · 2022
Cited alongside, same era.
What can transformers learn in-context? a case study of simple function classes
Shivam Garg, Dimitris Tsipras, Percy S Liang, and Gregory Valiant · 2022
Cited alongside, same era.
It’s raw! audio generation with state-space models
Karan Goel, Albert Gu, Chris Donahue, and Christopher Ré · 2022
Cited alongside, same era.
Efficiently modeling long sequences with structured state spaces
Transformers are ssms: Generalized models and efficient algorithms through structured state space duality
Tri Dao and Albert Gu · 2024
Closest in time.
Qlora: Efficient finetuning of quantized llms
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer · 2024
Closest in time.
Olmo: Accelerating the science of language models
Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, et al · 2024
Closest in time.
Mamba Precision Guidance
Albert Gu and Tri Dao · 2024
Closest in time.
Mamba PEFT
Huggingface · 2024
Closest in time.
Is attention required for icl? exploring the relationship between model architecture and in-context learning ability
Ivan Lee, Nan Jiang, and Taylor Berg-Kirkpatrick · 2024
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Albert Gu, Karan Goel, and Christopher Re · 2022
Cited alongside, same era.
Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning
Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel · 2022
Cited alongside, same era.
On the difficulty of learning chaotic dynamics with rnns
Jonas Mikhaeil, Zahra Monfared, and Daniel Durstewitz · 2022
Cited alongside, same era.
Einops: Clear and reliable tensor manipulations with einstein-like notation
Alex Rogozhnikov · 2022
Cited alongside, same era.
On lyapunov exponents for rnns: Understanding information propagation using dynamical systems tools
Ryan Vogt, Maximilian Puelma Touzel, Eli Shlizerman, and Guillaume Lajoie · 2022
Cited alongside, same era.
Emergent abilities of large language models
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al · 2022
Cited alongside, same era.
Pythia: A suite for analyzing large language models across training and scaling
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al · 2023
Cited alongside, same era.
Closest in time.
Spmamba: State-space model is all you need in speech separation
Kai Li and Guo Chen · 2024
Closest in time.
Stg-mamba: Spatial-temporal graph learning via selective state space model
Lincan Li, Hanchen Wang, Wenjie Zhang, and Adelle Coster · 2024
Closest in time.
Jamba: A hybrid transformer-mamba language model
Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, et al · 2024
Closest in time.
Vmamba: Visual state space model
Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, and Yunfan Liu · 2024
Closest in time.
Openelm: An efficient language model family with open-source training and inference framework
Sachin Mehta, Mohammad Hossein Sekhavat, Qingqing Cao, Maxwell Horton, Yanzi Jin, Chenfan Sun, Iman Mirzadeh, Mahyar Najibi, Dmitry Belenko, Peter Zatloukal, et al · 2024
Closest in time.
Can mamba learn how to learn? a comparative study on in-context learning tasks
Jongho Park, Jaeseung Park, Zheyang Xiong, Nayoung Lee, Jaewoong Cho, Samet Oymak, Kangwook Lee, and Dimitris Papailiopoulos · 2024
Closest in time.
Multichannel long-term streaming neural speech enhancement for static and moving speakers
Changsheng Quan and Xiaofei Li · 2024
Closest in time.
Openhermes
Teknium · 2024
Closest in time.
Promamba: Prompt-mamba for polyp segmentation
Jianhao Xie, Ruofan Liao, Ziang Zhang, Sida Yi, Yuesheng Zhu, and Guibo Luo · 2024
Closest in time.
Tinyllama: An open-source small language model
Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu · 2024
Closest in time.
Lima: Less is more for alignment
Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al · 2024
Closest in time.
Mambapeft: Exploring parameter-efficient fine-tuning for mamba
Masakazu Yoshimura, Teruaki Hayashi, and Yota Maeda · 2025
Closest in time.