HDMoLE: Mixture of LoRA Experts with Hierarchical Routing and Dynamic Thresholds for Fine-Tuning LLM-based ASR Models

S. Dou, E. Zhou, Y. Liu, S. Gao, W. Shen, L. Xiong, Y. Zhou, X. Wang, Z. Xi, X. Fan, S. Pu, J. Zhu, R. Zheng, T. Gui, Q. Zhang, and X. Huang, “LoRAMoE: Alleviating World Knowledge Forgetting in Large Language Models via MoE-Style Plugin,” in Proc. ACL , 2024, pp. 1932–1945

1945

Earlier work this paper cites.

N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. V. Le, G. E. Hinton, and J. Dean, “Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer,” in Proc. ICLR , 2017

2017

Earlier work this paper cites.

H. Bu, J. Du, X. Na, B. Wu, and H. Zheng, “AISHELL-1: An open-source Mandarin speech corpus and a speech recognition baseline,” in Proc. O-COCOSDA , 2017, pp. 1–5

2017

Earlier work this paper cites.

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is All you Need,” in Proc. NeurIPS , 2017, pp. 5998–6008

2017

Earlier work this paper cites.

J. Du, X. Na, X. Liu, and H. Bu, “AISHELL-2: Transforming Mandarin ASR Research Into Industrial Scale,” arXiv preprint arXiv:1808.10583 , 2018

Original

2018

Earlier work this paper cites.

N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. de Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly, “Parameter-Efficient Transfer Learning for NLP,” in Proc. ICML , 2019, pp. 2790–2799

2019

Earlier work this paper cites.

A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu, and R. Pang, “Conformer: Convolution-augmented Transformer for Speech Recognition,” in Proc. Interspeech , 2020, pp. 5036–5040

2020

Earlier work this paper cites.

X. L. Li and P. Liang, “Prefix-Tuning: Optimizing Continuous Prompts for Generation,” in Proc. ACL/IJCNLP , 2021, pp. 4582–4597

2021

Earlier work this paper cites.

B. Lester, R. Al-Rfou, and N. Constant, “The Power of Scale for Parameter-Efficient Prompt Tuning,” in Proc. EMNLP , 2021, pp. 3045–3059

2021

Earlier work this paper cites.

Z. Tang, D. Wang, Y. Xu, J. Sun, X. Lei, S. Zhao, C. Wen, X. Tan et al. , “KeSpeech: An Open Source Speech Dataset of Mandarin and Its Eight Subdialects,” in Proc. NeurIPS Datasets and Benchmarks Track , 2021

2021

Earlier work this paper cites.

Y. Fu, L. Cheng, S. Lv, Y. Jv, Y. Kong, Z. Chen, Y. Hu, L. Xie, J. Wu, H. Bu, X. Xu, J. Du, and J. Chen, “AISHELL-4: an open source dataset for speech enhancement, separation, recognition and speaker diarization in conference scenario,” in Proc. Interspeech , 2021, pp. 3665–3669

2021

Earlier work this paper cites.

L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. F. Christiano, J. Leike, and R. Lowe, “Training language models to follow instructions with human feedback,” in Proc. NeurIPS , 2022

2022

Earlier work this paper cites.

H. Liu, D. Tam, M. Muqeeth, J. Mohta, T. Huang, M. Bansal, and C. Raffel, “Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning,” in Proc. NeurIPS , 2022

2022

Cited alongside, same era.

E. B. Zaken, Y. Goldberg, and S. Ravfogel, “BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models,” in Proc. ACL , 2022, pp. 1–9

2022

Cited alongside, same era.

E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-Rank Adaptation of Large Language Models,” in Proc. ICLR , 2022

2022

Cited alongside, same era.

W. Fedus, J. Dean, and B. Zoph, “A Review of Sparse Expert Models in Deep Learning,” arXiv preprint arXiv:2209.01667 , 2022

Original

2022

Cited alongside, same era.

B. Zoph, I. Bello, S. Kumar, N. Du, Y. Huang, J. Dean, N. Shazeer, and W. Fedus, “ST-MoE: Designing Stable and Transferable Sparse Expert Models,” arXiv preprint arXiv:2202.08906 , 2022

A. Yang, B. Yang, B. Hui, B. Zheng, B. Yu, C. Zhou, C. Li, C. Li, D. Liu, F. Huang et al. , “Qwen2 Technical Report,” arXiv preprint arXiv:2407.10671 , 2024

Original

2024

Closest in time.

Y. Chu, J. Xu, Q. Yang, H. Wei, X. Wei, Z. Guo, Y. Leng, Y. Lv, J. He, J. Lin et al. , “Qwen2-Audio Technical Report,” arXiv preprint arXiv:2407.10759 , 2024

Original

2024

Closest in time.

Y. Bai, J. Chen, J. Chen, W. Chen, Z. Chen, C. Ding, L. Dong, Q. Dong, Y. Du, K. Gao et al. , “Seed-ASR: Understanding Diverse Speech and Contexts with LLM-based Speech Recognition,” arXiv preprint arXiv:2407.04675 , 2024

Original

2024

Closest in time.

X. Geng, T. Xu, K. Wei, B. Mu, H. Xue, H. Wang, Y. Li, P. Guo, Y. Dai, L. Li, M. Shao, and L. Xie, “Unveiling the Potential of LLM-Based ASR on Chinese Open-Source Datasets,” arXiv preprint arXiv:2405.02132 , 2024

Original

2024

Closest in time.