Fetching the paper…
Reading the bibliography…
Selective state-space models (SSMs) like Mamba overcome some of the shortcomings of Transformers, such as quadratic computational complexity with sequence length and large inference-time memory requirements from the key-value cache.
“Language Models are Few-shot Learners”
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry and Amanda Askell · 1901
Earlier work this paper cites.
“Learning Question Classifiers”
Xin Li and Dan Roth · 2002
Earlier work this paper cites.
“Neural Machine Translation by Jointly Learning to Align and Translate”
Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio · 2014
Earlier work this paper cites.
Jimmy Ba, Jamie Kiros and Geoffrey Hinton · 2016
Earlier work this paper cites.
“Gaussian Error Linear Units (GELUs)”
Dan Hendrycks and Kevin Gimpel · 2016
Earlier work this paper cites.
“TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension”
Mandar Joshi, Eunsol Choi, Daniel Weld and Luke Zettlemoyer · 2017
Earlier work this paper cites.
“RACE: Large-scale ReAding Comprehension Dataset From Examinations”
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang and Eduard Hovy · 2017
Earlier work this paper cites.
“Attention is All You Need”
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Łukasz Kaiser and Illia Polosukhin · 2017
Earlier work this paper cites.
“Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge”
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick and Oyvind Tafjord · 2018
Earlier work this paper cites.
“The NarrativeQA Reading Comprehension Challenge”
Tomáš Kočiskỳ, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Hermann, Gábor Melis and Edward Grefenstette · 2018
Earlier work this paper cites.
Taku Kudo and John Richardson · 2018
Earlier work this paper cites.
“Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering”
Todor Mihaylov, Peter Clark, Tushar Khot and Ashish Sabharwal · 2018
Earlier work this paper cites.
“Know what you don’t Know: Unanswerable Questions for SQuAD”
Pranav Rajpurkar, Robin Jia and Percy Liang · 2018
Earlier work this paper cites.
“HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering”
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov and Christopher Manning · 2018
Earlier work this paper cites.
“PubMedQA: A Dataset for Biomedical Research Question Answering”
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen and Xinghua Lu · 2019
Earlier work this paper cites.
“Latent Retrieval for Weakly Supervised Open Domain Question Answering”
Kenton Lee, Ming-Wei Chang and Kristina Toutanova · 2019
Earlier work this paper cites.
“Megatron-LM: Training Multi-billion Parameter Language Models using Model Parallelism”
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper and Bryan Catanzaro · 2019
Earlier work this paper cites.
“HellaSwag: Can a Machine Really Finish your Sentence?”
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi and Yejin Choi · 2019
Earlier work this paper cites.
“Root Mean Square Layer Normalization”
Biao Zhang and Rico Sennrich · 2019
Cited alongside, same era.
“PIQA: Reasoning about Physical Commonsense in Natural Language”
Yonatan Bisk, Rowan Zellers, Jianfeng Gao and Yejin Choi · 2020
Cited alongside, same era.
“Measuring Massive Multitask Language Understanding”
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song and Jacob Steinhardt · 2020
Cited alongside, same era.
“Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps”
Xanh Ho, Anh-Khoa Nguyen, Saku Sugawara and Akiko Aizawa · 2020
Cited alongside, same era.
“GLU Variants Improve Transformer”
Noam Shazeer · 2020
Cited alongside, same era.
“A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers”
“NTK-aware Scaled RoPE allows LLaMA models to have Extended (8k+) Context Size Without any Fine-tuning and Minimal Perplexity Degradation”, 2023
bloc97 · 2023
Later among the works it cites.
“A Framework for Few-shot Language Model Evaluation”
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang and Andy Zou · 2023
Later among the works it cites.
“Mamba: Linear-time Sequence Modeling with Selective State Spaces”
Albert Gu and Tri Dao · 2023
Later among the works it cites.
Albert Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Chaplot, Diego Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample and Lucile Saulnier · 2023
Later among the works it cites.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah Smith and Matt Gardner · 2021
Cited alongside, same era.
“Efficiently Modeling Long Sequences with Structured State Spaces”
Albert Gu, Karan Goel and Christopher Re · 2021
Cited alongside, same era.
“TruthfulQA: Measuring How Models Mimic Human Falsehoods”
Stephanie Lin, Jacob Hilton and Owain Evans · 2021
Cited alongside, same era.
“Efficient Large-scale Language Model Training on GPU Clusters using Megatron-LM”
Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer and Bryan Catanzaro · 2021
Cited alongside, same era.
“WinoGrande: An Adversarial Winograd Schema Challenge at Scale”
Keisuke Sakaguchi, Ronan Bras, Chandra Bhagavatula and Yejin Choi · 2021
Cited alongside, same era.
“Reducing Activation Recomputation in Large Transformer Models”
Vijay Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi and Bryan Catanzaro · 2022
Cited alongside, same era.
“Scrolls: Standardized Comparison over Long Language Sequences”
Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva and Jonathan Berant · 2022
Cited alongside, same era.
NVIDIA · 2023
Later among the works it cites.
“Diagonal State Space Augmented Transformers for Speech Recognition”
George Saon, Ankit Gupta and Xiaodong Cui · 2023
Later among the works it cites.
“Llama 2: Open Foundation and Fine-tuned Chat Models”
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava and Shruti Bhosale · 2023
Later among the works it cites.
“Effective Long-context Scaling of Foundation Models”
Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Sankararaman and Barlas Oguz · 2023
Later among the works it cites.
“Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality”
Tri Dao and Albert Gu · 2024
Closest in time.
“Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models”
Soham De, Samuel Smith, Anushan Fernando, Aleksandar Botev, George Cristian-Muraru, Albert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen and Srivatsan Srinivasan · 2024
Closest in time.
“Zamba: A Compact 7B SSM Hybrid Model”
Paolo Glorioso, Quentin Anthony, Yury Tokpanov, James Whittington, Jonathan Pilault, Adam Ibrahim and Beren Millidge · 2024
Closest in time.
“RULER: What’s the Real Context Size of Your Long-Context Language Models?”
Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia and Boris Ginsburg · 2024
Closest in time.
“Repeat After Me: Transformers are Better than State Space Models at Copying”
Samy Jelassi, David Brandfonbrener, Sham Kakade and Eran Malach · 2024
Closest in time.
“Jamba: A Hybrid Transformer-mamba Language Model”
Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov and Shai Shalev-Shwartz · 2024
Closest in time.
“Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Tasks”
Jongho Park, Jaeseung Park, Zheyang Xiong, Nayoung Lee, Jaewoong Cho, Samet Oymak, Kangwook Lee and Dimitris Papailiopoulos · 2024
Closest in time.
“Nemotron-4 15B Technical Report”
Jupinder Parmar, Shrimai Prabhumoye, Joseph Jennings, Mostofa Patwary, Sandeep Subramanian, Dan Su, Chen Zhu, Deepak Narayanan, Aastha Jhunjhunwala and Ayush Dattagupta · 2024
Closest in time.
“Block-state Transformers”
Jonathan Pilault, Mahan Fathi, Orhan Firat, Chris Pal, Pierre-Luc Bacon and Ross Goroshin · 2024
Closest in time.
“Roformer: Enhanced Transformer with Rotary Position Embedding”
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo and Yunfeng Liu · 2024
Closest in time.