Fetching the paper…

Reading the bibliography…

2024

DoTA: Weight-Decomposed Tensor Adaptation for Large Language Models

Hu, Xiaolin, Cheng, Xiang, Liu, Peiyu et al.

Original
PDF
Post on X

Understand

Low-rank adaptation (LoRA) reduces the computational and memory demands of fine-tuning large language models (LLMs) by approximating updates with low-rank matrices.

However, low-rank approximation in two-dimensional space fails to capture high-dimensional structures within the target matrix.
Recently, tensor decomposition methods have been explored for fine-tuning LLMs, leveraging their ability to extract structured information.
Yet, these approaches primarily rely on random initialization, and the impact of initialization on tensor adaptation remains underexplored.

DoTA: Weight-Decomposed Tensor Adaptation for Large Language Models · Around

Built on

De Silva, V., Lim, L.H.: Tensor rank and the ill-posedness of the best low-rank approximation problem. SIAM Journal on Matrix Analysis and Applications 30
2008
Earlier work this paper cites.
Pirvu, B., Murg, V., Cirac, J.I., Verstraete, F.: Matrix product operator representations. New Journal of Physics 12
2010
Earlier work this paper cites.
Li, C., Farkhoor, H., Liu, R., Yosinski, J.: Measuring the intrinsic dimension of objective landscapes. arXiv preprint arXiv:1804.08838 (2018)
Original
2018
Earlier work this paper cites.
Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Burstein, J., Doran, C., Solorio, T. (eds.) Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers). pp. 4171–4186. Association for Computational Linguistics (2019)
2019
Earlier work this paper cites.
Gao, Z.F., Cheng, S., He, R.Q., Xie, Z., Zhao, H.H., Lu, Z.Y., Xiang, T.: Compressing deep neural networks by matrix product operators. Physical Review Research 2
2020
Earlier work this paper cites.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al.: Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 (2021)
Original
2021
Earlier work this paper cites.
He, J., Zhou, C., Ma, X., Berg-Kirkpatrick, T., Neubig, G.: Towards a unified view of parameter-efficient transfer learning. arXiv preprint arXiv:2110.04366 (2021)
Original
2021
Earlier work this paper cites.
Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W.: Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 (2021)
Original
2021
Earlier work this paper cites.
Yin, C., Acun, B., Liu, X., Wu, C.J.: TT-Rec: Tensor Train Compression for Deep Learning Recommendation Models (Jan 2021), arXiv:2101.11714 [cs]
Original
2021
Earlier work this paper cites.
Bommasani, R., Hudson, D.A., Adeli, E., Altman, e.a.: On the Opportunities and Risks of Foundation Models (Jul 2022), arXiv:2108.07258 [cs]
Original
2022
Earlier work this paper cites.

Similar

Hu, Z., Wang, L., Lan, Y., Xu, W., Lim, E.P., Bing, L., Xu, X., Poria, S., Lee, R.K.W.: Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models. arXiv preprint arXiv:2304.01933 (2023)
Original
2023
Cited alongside, same era.
Kalajdzievski, D.: A rank stabilization scaling factor for fine-tuning with lora. arXiv preprint arXiv:2312.03732 (2023)
Original
2023
Cited alongside, same era.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al.: Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023)
Original
2023
Cited alongside, same era.
Yu, L., Jiang, W., Shi, H., Yu, J., Liu, Z., Zhang, Y., Kwok, J.T., Li, Z., Weller, A., Liu, W.: Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309.12284 (2023)
Original
2023
Cited alongside, same era.

Then

Hayou, S., Ghosh, N., Yu, B.: LoRA+: Efficient Low Rank Adaptation of Large Models (Jul 2024), arXiv:2402.12354 [cs, stat]
Original
2024
Closest in time.
Huang, M., Shen, A., Li, K., Peng, H., Li, B., Yu, H.: EdgeLLM: A Highly Efficient CPU-FPGA Heterogeneous Edge Accelerator for Large Language Models (Jul 2024), arXiv:2407.21325 [cs]
Original
2024
Closest in time.
Liu, S.Y., Wang, C.Y., Yin, H., Molchanov, P., Wang, Y.C.F., Cheng, K.T., Chen, M.H.: Dora: Weight-decomposed low-rank adaptation. arXiv preprint arXiv:2402.09353 (2024)
Original
2024
Closest in time.
Meng, F., Wang, Z., Zhang, M.: Pissa: Principal singular values and singular vectors adaptation of large language models. arXiv preprint arXiv:2404.02948 (2024)
Original
2024
Closest in time.
Qiu, S., Potapczynski, A., Finzi, M., Goldblum, M., Wilson, A.G.: Compute Better Spent: Replacing Dense Layers with Structured Matrices (Jun 2024), arXiv:2406.06248 [cs]

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Zhang, Q., Chen, M., Bukharin, A., Karampatziakis, N., He, P., Cheng, Y., Chen, W., Zhao, T.: AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning (Mar 2023)

2023

Cited alongside, same era.

Bershatsky, D., Cherniuk, D., Daulbaev, T., Oseledets, I.: Lotr: Low tensor rank weight adaptation. arXiv preprint arXiv:2402.01376 (2024)

2024

Cited alongside, same era.

Chen, Z., Dangovski, R., Loh, C., Dugan, O., Luo, D., Soljačić, M.: Quanta: Efficient high-rank fine-tuning of llms with quantum-informed tensor adaptation. arXiv preprint arXiv:2406.00132 (2024)

2024

Cited alongside, same era.

Dettmers, T., Pagnoni, A., Holtzman, A., Zettlemoyer, L.: Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems 36

2024

Cited alongside, same era.

Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al.: The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024)

2024

Cited alongside, same era.

Gao, Z.F., Liu, P., Zhao, W.X., Xie, Z.Y., Wen, J.R., Lu, Z.Y.: Compression image dataset based on multiple matrix product states. In: Arai, K. (ed.) Advances in information and communication. pp. 621–638. Springer Nature Switzerland, Cham (2024)

2024

Cited alongside, same era.

Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I.: Language Models are Unsupervised Multitask Learners

Cited in the paper.

2024

Closest in time.

Si, C., Wang, X., Yang, X., Xu, Z., Li, Q., Dai, J., Qiao, Y., Yang, X., Shen, W.: FLoRA: Low-Rank Core Space for N-dimension (May 2024), arXiv:2405.14739 [cs]

2024

Closest in time.

Wang, H., Xiao, Z., Li, Y., Wang, S., Chen, G., Chen, Y.: MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning (Jun 2024), arXiv:2406.09044 [cs]

2024

Closest in time.

Wang, S., Yu, L., Li, J.: LoRA-GA: Low-Rank Adaptation with Gradient Approximation (Jul 2024), arXiv:2407.05000 [cs]

2024

Closest in time.

Yang, Y., Zhou, J., Wong, N., Zhang, Z.: LoRETTA: Low-Rank Economic Tensor-Train Adaptation for Ultra-Low-Parameter Fine-Tuning of Large Language Models (Feb 2024), arXiv:2402.11417 [cs]

2024

Closest in time.