Fetching the paper…
Reading the bibliography…
Large Language Models' (LLMs) weight matrices can often be expressed in low-rank form with potential to relax memory and compute resource requirements.
Revealing the structure of deep neural networks via convex duality, 2021
Ergen, T. and Pilanci, M · 2002
Earlier work this paper cites.
Directional convergence and alignment in deep learning, 2020
Ji, Z. and Telgarsky, M · 2006
Earlier work this paper cites.
Understanding self-supervised learning with dual deep networks, 2021
Tian, Y., Yu, L., Chen, X., and Ganguli, S · 2010
Earlier work this paper cites.
A neural network for factoid question answering over paragraphs
Iyyer, M., Boyd-Graber, J. L., Claudino, L. M. B., Socher, R., and Daumé, H · 2014
Earlier work this paper cites.
A thorough examination of the cnn/daily mail reading comprehension task
Chen, D., Bolton, J., and Manning, C. D · 2016
Earlier work this paper cites.
Freezeout: Accelerate training by progressively freezing layers
Brock, A., Lim, T., Ritchie, J. M., and Weston, N · 2017
Earlier work this paper cites.
Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension
Joshi, M., Choi, E., Weld, D. S., and Zettlemoyer, L · 2017
Earlier work this paper cites.
Parameter-efficient transfer learning for nlp
Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M., and Gelly, S · 2019
Earlier work this paper cites.
Freebaseqa: A new factoid qa data set matching trivia-style question-answer pairs with freebase
Jiang, K., Wu, D., and Jiang, H · 2019
Earlier work this paper cites.
Warp: Word-level adversarial reprogramming
Hambardzumyan, K., Khachatrian, H., and May, J · 2021
Earlier work this paper cites.
Autofreeze: Automatically freezing model blocks to accelerate fine-tuning
Liu, Y., Agarwal, S., and Venkataraman, S · 2021
Earlier work this paper cites.
Factual probing is [mask]: Learning vs. learning to recall
Zhong, Z., Friedman, D., and Chen, D · 2021
Earlier work this paper cites.
Black-box prompt learning for pre-trained language models
Diao, S., Huang, Z., Xu, R., Li, X., Lin, Y., Zhou, X., and Zhang, T · 2022
Earlier work this paper cites.
Training invariances and the low-rank phenomenon: beyond linear networks, 2022
Le, T. and Jegelka, S · 2022
Cited alongside, same era.
The role of linear layers in nonlinear interpolating networks, 2022
Ongie, G. and Willett, R · 2022
Cited alongside, same era.
Implicit regularization towards rank minimization in relu networks, 2022
Timor, N., Vardi, G., and Shamir, O · 2022
Cited alongside, same era.
Qlora: Efficient finetuning of quantized llms, 2023
Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L · 2023
Cited alongside, same era.
Sparsegpt: Massive language models can be accurately pruned in one-shot
Frantar, E. and Alistarh, D · 2023
Asvd: Activation-aware singular value decomposition for compressing large language models
Yuan, Z., Shang, Y., Song, Y., Wu, Q., Yan, Y., and Sun, G · 2023
Later among the works it cites.
Lora-fa: Memory-efficient low-rank adaptation for large language models fine-tuning
Zhang, L., Zhang, L., Shi, S., Chu, X., and Li, B · 2023
Later among the works it cites.
Inrank: Incremental low-rank learning
Zhao, J., Zhang, Y., Chen, B., Schäfer, F., and Anandkumar, A · 2023
Later among the works it cites.
Judging llm-as-a-judge with mt-bench and chatbot arena
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al · 2023
Later among the works it cites.
Sgd and weight decay secretly minimize the rank of your neural network, 2024
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Cited alongside, same era.
Multi-dimensional evaluation of text summarization with in-context learning
Jain, S., Keshava, V., Sathyendra, S. M., Fernandes, P., Liu, P., Neubig, G., and Zhou, C · 2023
Cited alongside, same era.
Compressing llms: The truth is rarely pure and never simple
Jaiswal, A., Gan, Z., Du, X., Zhang, B., Wang, Z., and Yang, Y · 2023
Cited alongside, same era.
Li, Y., Yu, Y., Zhang, Q., Liang, C., He, P., Chen, W., and Zhao, T · 2023
Cited alongside, same era.
Tied-lora: Enhacing parameter efficiency of lora with weight tying
Renduchintala, A., Konuk, T., and Kuchaiev, O · 2023
Cited alongside, same era.
Matrix compression via randomized low rank and low precision factorization
Saha, R., Srivastava, V., and Pilanci, M · 2023
Cited alongside, same era.
S-lora: Serving thousands of concurrent lora adapters
Sheng, Y., Cao, S., Li, D., Hooper, C., Lee, N., Yang, S., Chou, C., Zhu, B., Zheng, L., Keutzer, K., et al · 2023
Cited alongside, same era.
A simple and effective pruning approach for large language models
Sun, M., Liu, Z., Bair, A., and Kolter, J. Z · 2023
Cited alongside, same era.
Galanti, T., Siegel, Z. S., Gupte, A., and Poggio, T · 2024
Closest in time.
Flora: Low-rank adapters are secretly gradient compressors
Hao, Y., Cao, Y., and Mou, L · 2024
Closest in time.
Lora+: Efficient low rank adaptation of large models
Hayou, S., Ghosh, N., and Yu, B · 2024
Closest in time.
Compressing llms: The truth is rarely pure and never simple, 2024
Jaiswal, A., Gan, Z., Du, X., Zhang, B., Wang, Z., and Yang, Y · 2024
Closest in time.
Smartfrz: An efficient training framework using attention-based layer freezing
Li, S., Yuan, G., Dai, Y., Zhang, Y., Wang, Y., and Tang, X · 2024
Closest in time.
Dora: Weight-decomposed low-rank adaptation
Liu, S.-Y., Wang, C.-Y., Yin, H., Molchanov, P., Wang, Y.-C. F., Cheng, K.-T., and Chen, M.-H · 2024
Closest in time.
Pissa: Principal singular values and singular vectors adaptation of large language models, 2024
Meng, F., Wang, Z., and Zhang, M · 2024
Closest in time.
Svd-llm: Truncation-aware singular value decomposition for large language model compression
Wang, X., Zheng, Y., Wan, Z., and Zhang, M · 2024
Closest in time.
Chain of lora: Efficient fine-tuning of language models via residual learning
Xia, W., Qin, C., and Hazan, E · 2024
Closest in time.