Built on

Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistilBERT, a Distilled Version of BERT: Smaller, Faster, Cheaper and Lighter. arXiv preprint arXiv:1910.01108
Original
1910
Earlier work this paper cites.
Noam Shazeer. 2019. Fast Transformer Decoding: One Write-Head is All You Need. arXiv preprint arXiv:1911.02150
Original
1911
Earlier work this paper cites.
Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. 2020. Linformer: Self-Attention with Linear Complexity. arXiv preprint arXiv:2006.04768
Original
2006
Earlier work this paper cites.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. In Advances in Neural Information Processing Systems 30
2017
Earlier work this paper cites.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-Attention with Relative Position Representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL 2018)
2018
Earlier work this paper cites.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Pranav Shyam, Girish Sastry, Amanda Askell, and others. 2020. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems 33
2020
Earlier work this paper cites.
Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The Efficient Transformer. In International Conference on Learning Representations (ICLR 2020)
2020
Earlier work this paper cites.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A Lite BERT for Self- Supervised Learning of Language Representations. In International Conference on Learning Representations (ICLR 2020)
2020
Earlier work this paper cites.
Rishi Bommasani et al. 2021. On the Opportunities and Risks of Foundation Models. arXiv preprint arXiv:2108.07258
Original
2021
Earlier work this paper cites.
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, and others. 2021. Rethinking Attention with Performers. In International Conference on Learning Representations (ICLR 2021)
2021
Earlier work this paper cites.

Similar

Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. 2021. RoFormer: Enhanced Transformer with Rotary Position Embedding. In Proceedings of ACL-IJCNLP 2021
2021
Cited alongside, same era.
Ben Wang and Aran Komatsuzaki. 2021. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. (2021)
2021
Cited alongside, same era.
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. In Advances in Neural Information Processing Systems 35
2022
Cited alongside, same era.
Jordan Hoffmann et al. 2022. Training Compute- Optimal Large Language Models. arXiv preprint arXiv:2203.15556
Original
2022
Cited alongside, same era.
Ofir Press, Noah A. Smith, and Mike Lewis. 2022. Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation. arXiv preprint arXiv:2108.12409

Then

Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. 2023. YARN: Efficient Context Window Extension of Large Language Models. arXiv preprint arXiv:2309.00071
Original
2023
Later among the works it cites.
Hugo Touvron et al. 2023. Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv preprint arXiv:2307.09288
Original
2023
Later among the works it cites.
DeepSeek-AI. 2024. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model. arXiv preprint arXiv:2405.04434
Original
2024
Later among the works it cites.
Olga Golovneva, Tianlu Wang, Jason Weston, and Sainbayar Sukhbaatar. 2024. Contextual Position Encoding: Learning to Count What’s Important. arXiv preprint arXiv:2405.18719
Original
2024
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2025

Latent Multi-Head Attention for Small Language Models

Mehta, Sushant, Dandekar, Raj, Dandekar, Rajat et al.

Understand

We present the first comprehensive study of latent multi-head attention (MLA) for small language models, revealing interesting efficiency-quality trade-offs.

Training 30M-parameter GPT models on 100,000 synthetic stories, we benchmark three architectural variants: standard multi-head attention (MHA), MLA, and MLA with rotary positional embeddings (MLA+RoPE).
Our key finding is that MLA+RoPE with half-rank latent dimensions (r = d/2) achieves a 45% KV-cache memory reduction while incurring only a 0.3% increase in validation loss (essentially matching MHA quality)- a Pareto improvement for memory constrained deployment.
We further show that RoPE is crucial for MLA in small models: without it, MLA underperforms vanilla attention by 3-5%, but with RoPE, it surpasses vanilla by 2%.

Built on

Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistilBERT, a Distilled Version of BERT: Smaller, Faster, Cheaper and Lighter. arXiv preprint arXiv:1910.01108
Original
1910
Earlier work this paper cites.
Noam Shazeer. 2019. Fast Transformer Decoding: One Write-Head is All You Need. arXiv preprint arXiv:1911.02150
Original
1911
Earlier work this paper cites.
Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. 2020. Linformer: Self-Attention with Linear Complexity. arXiv preprint arXiv:2006.04768
Original
2006
Earlier work this paper cites.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. In Advances in Neural Information Processing Systems 30
2017
Earlier work this paper cites.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-Attention with Relative Position Representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL 2018)
2018
Earlier work this paper cites.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Pranav Shyam, Girish Sastry, Amanda Askell, and others. 2020. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems 33
2020
Earlier work this paper cites.
Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The Efficient Transformer. In International Conference on Learning Representations (ICLR 2020)
2020
Earlier work this paper cites.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A Lite BERT for Self- Supervised Learning of Language Representations. In International Conference on Learning Representations (ICLR 2020)
2020
Earlier work this paper cites.
Rishi Bommasani et al. 2021. On the Opportunities and Risks of Foundation Models. arXiv preprint arXiv:2108.07258
Original
2021
Earlier work this paper cites.
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, and others. 2021. Rethinking Attention with Performers. In International Conference on Learning Representations (ICLR 2021)
2021
Earlier work this paper cites.

Similar

Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. 2021. RoFormer: Enhanced Transformer with Rotary Position Embedding. In Proceedings of ACL-IJCNLP 2021
2021
Cited alongside, same era.
Ben Wang and Aran Komatsuzaki. 2021. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. (2021)
2021
Cited alongside, same era.
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. In Advances in Neural Information Processing Systems 35
2022
Cited alongside, same era.
Jordan Hoffmann et al. 2022. Training Compute- Optimal Large Language Models. arXiv preprint arXiv:2203.15556
Original
2022
Cited alongside, same era.
Ofir Press, Noah A. Smith, and Mike Lewis. 2022. Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation. arXiv preprint arXiv:2108.12409

Then

Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. 2023. YARN: Efficient Context Window Extension of Large Language Models. arXiv preprint arXiv:2309.00071
Original
2023
Later among the works it cites.
Hugo Touvron et al. 2023. Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv preprint arXiv:2307.09288
Original
2023
Later among the works it cites.
DeepSeek-AI. 2024. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model. arXiv preprint arXiv:2405.04434
Original
2024
Later among the works it cites.
Olga Golovneva, Tianlu Wang, Jason Weston, and Sainbayar Sukhbaatar. 2024. Contextual Position Encoding: Learning to Count What’s Important. arXiv preprint arXiv:2405.18719
Original
2024
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Original

2022

Cited alongside, same era.

Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. 2023. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL 2023)

2023

Cited alongside, same era.

Ronen Eldan and Yuanzhi Li. 2023. TinyStories: How Small Can Language Models Be and Still Produce Coherent Text? arXiv preprint arXiv:2305.07759

Original

2023

Cited alongside, same era.

Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment. arXiv preprint arXiv:2303.16634

Original

2023

Cited alongside, same era.

OpenAI. 2023. GPT-4 Technical Report. arXiv preprint arXiv:2303.08774

Original

2023

Cited alongside, same era.

MiniCPM Team. 2024. MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies. arXiv preprint arXiv:2404.06395

Original

2024

Later among the works it cites.

MobiLLM Team. 2024. MobiLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases. In International Conference on Machine Learning (ICML 2024)

2024

Later among the works it cites.

Fali Wang, Zhiwei Zhang, Xianren Zhang, Zongyu Wu, TzuHao Mo, Qiuhao Lu, Wanjing Wang, Rui Li, Junjie Xu, Xianfeng Tang, Qi He, Yao Ma, Ming Huang, and Suhang Wang. 2024. A Comprehensive Survey of Small Language Models in the Era of Large Language Models. arXiv preprint arXiv:2411.03350

Original

2024

Later among the works it cites.

Fanxu Meng, Yanan Li, and William Chan. 2025. TransMLA: Multi-Head Latent Attention Is All You Need. arXiv preprint arXiv:2502.07864

Original

2025

Closest in time.

Nirvan Patil, Malhar A. Inamdar, Agnivo Gosai, Guruprasad Pathak, Anish Joshi, and others. 2025. Regional Tiny Stories: Using Small Models to Compare Language Learning and Tokenizer Performance. arXiv preprint arXiv:2504.07989

Original

2025

Closest in time.