Built on

T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, “Language models are few-shot learners,” CoRR , vol. abs/2005.14165, 2020. [Online]. Available: https://arxiv.org/abs/2005.14165
Original
2005
Earlier work this paper cites.
D. P. Kingma, “Adam: A method for stochastic optimization,” Proceedings of ICLR , 2015
2015
Earlier work this paper cites.
W. Chan, N. Jaitly, Q. Le, and O. Vinyals, “Listen, attend and spell: A neural network for large vocabulary conversational speech recognition,” in 2016 IEEE international conference on acoustics, speech and signal processing (ICASSP) . IEEE, 2016, pp. 4960–4964
2016
Earlier work this paper cites.
M. Post, “A call for clarity in reporting bleu scores,” arXiv preprint arXiv:1804.08771 , 2018
Original
2018
Earlier work this paper cites.
O. Kuchaiev, J. Li, H. Nguyen, O. Hrinchuk, R. Leary, B. Ginsburg, S. Kriman, S. Beliaev, V. Lavrukhin, J. Cook et al. , “Nemo: a toolkit for building ai applications using neural modules,” arXiv preprint arXiv:1909.09577 , 2019
Original
2019
Earlier work this paper cites.
B. Zhang and R. Sennrich, “Root mean square layer normalization,” Proceedings of NeurIPS , 2019
2019
Earlier work this paper cites.
C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” Journal of Machine Learning Research , 2020
2020
Earlier work this paper cites.
Y. Liu, J. Gu, N. Goyal, X. Li, S. Edunov, M. Ghazvininejad, M. Lewis, and L. Zettlemoyer, “Multilingual denoising pre-training for neural machine translation,” in Transactions of the Association for Computational Linguistics . MIT Press, 2020, pp. 726–742
2020
Earlier work this paper cites.
M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer, “Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics . ACL, 2020, pp. 7871–7880
2020
Earlier work this paper cites.
G. Ramírez-Sánchez, J. Zaragoza-Bernabeu, M. Bañón, and S. Ortiz-Rojas, “Bifixer and bicleaner: two open-source tools to clean your parallel data.” in Proceedings of the 22nd Annual Conference of the European Association for Machine Translation . European Association for Machine Translation, 2020, pp. 291–298
2020
Earlier work this paper cites.
R. Zheng, J. Chen, M. Ma, and L. Huang, “Fused acoustic and text encoding for multimodal bilingual pretraining and speech translation,” in International Conference on Machine Learning . PMLR, 2021, pp. 12 736–12 746
2021
Earlier work this paper cites.

Similar

P. Żelasko, D. Povey, J. Trmal, and S. Khudanpur, “Lhotse: a speech data representation library for the modern deep learning ecosystem,” Proceedings of NeurIPS Workshop on Data-Centric AI , 2021
2021
Cited alongside, same era.
J.-B. Alayrac et al. , “Flamingo: a visual language model for few-shot learning,” Advances in Neural Information Processing Systems , vol. 35, pp. 23 716–23 736, 2022
2022
Cited alongside, same era.
Y. Tang, H. Gong, N. Dong, C. Wang, W.-N. Hsu, J. Gu, A. Baevski, X. Li, A. Mohamed, M. Auli et al. , “Unified speech-text pre-training for speech translation and recognition,” arXiv preprint arXiv:2204.05409 , 2022
Original
2022
Cited alongside, same era.
A. Bapna, C. Cherry, Y. Zhang, Y. Jia, M. Johnson, Y. Cheng, S. Khanuja, J. Riesa, and A. Conneau, “mslam: Massively multilingual joint pre-training for speech and text,” arXiv preprint arXiv:2202.01374 , 2022
Original
2022
Cited alongside, same era.

Then

S. Mittal, O. Hrinchuk, and O. Kuchaiev, “Leveraging synthetic targets for machine translation,” 2023. [Online]. Available: https://arxiv.org/abs/2305.06155
Original
2023
Later among the works it cites.
T. Nguyen, C. V. Nguyen, V. D. Lai, H. Man, N. T. Ngo, F. Dernoncourt, R. A. Rossi, and T. H. Nguyen, “Culturax: A cleaned, enormous, and multilingual dataset for large language models in 167 languages,” 2023. [Online]. Available: https://arxiv.org/abs/2309.09400
Original
2023
Later among the works it cites.
A. Conneau, M. Ma, S. Khanuja, Y. Zhang, V. Axelrod, S. Dalmia, J. Riesa, C. Rivera, and A. Bapna, “Fleurs: Few-shot learning evaluation of universal representations of speech,” in 2022 IEEE Spoken Language Technology Workshop (SLT) . IEEE, 2023, pp. 798–805
2023
Later among the works it cites.
L. Barrault, Y.-A. Chung, M. C. Meglioli, D. Dale, N. Dong, M. Duppenthaler, P.-A. Duquenne, B. Ellis, H. Elsahar, J. Haaheim et al. , “Seamless: Multilingual expressive and streaming speech translation,” arXiv preprint arXiv:2312.05187 , 2023
Original

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2024

EMMeTT: Efficient Multimodal Machine Translation Training

Żelasko, Piotr, Chen, Zhehuai, Wang, Mengru et al.

Understand

A rising interest in the modality extension of foundation language models warrants discussion on the most effective, and efficient, multimodal training approach.

This work focuses on neural machine translation (NMT) and proposes a joint multimodal training regime of Speech-LLM to include automatic speech translation (AST).
We investigate two different foundation model architectures, decoder-only GPT and encoder-decoder T5, extended with Canary-1B's speech encoder.
To handle joint multimodal training, we propose a novel training framework called EMMeTT.

Built on

T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, “Language models are few-shot learners,” CoRR , vol. abs/2005.14165, 2020. [Online]. Available: https://arxiv.org/abs/2005.14165
Original
2005
Earlier work this paper cites.
D. P. Kingma, “Adam: A method for stochastic optimization,” Proceedings of ICLR , 2015
2015
Earlier work this paper cites.
W. Chan, N. Jaitly, Q. Le, and O. Vinyals, “Listen, attend and spell: A neural network for large vocabulary conversational speech recognition,” in 2016 IEEE international conference on acoustics, speech and signal processing (ICASSP) . IEEE, 2016, pp. 4960–4964
2016
Earlier work this paper cites.
M. Post, “A call for clarity in reporting bleu scores,” arXiv preprint arXiv:1804.08771 , 2018
Original
2018
Earlier work this paper cites.
O. Kuchaiev, J. Li, H. Nguyen, O. Hrinchuk, R. Leary, B. Ginsburg, S. Kriman, S. Beliaev, V. Lavrukhin, J. Cook et al. , “Nemo: a toolkit for building ai applications using neural modules,” arXiv preprint arXiv:1909.09577 , 2019
Original
2019
Earlier work this paper cites.
B. Zhang and R. Sennrich, “Root mean square layer normalization,” Proceedings of NeurIPS , 2019
2019
Earlier work this paper cites.
C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” Journal of Machine Learning Research , 2020
2020
Earlier work this paper cites.
Y. Liu, J. Gu, N. Goyal, X. Li, S. Edunov, M. Ghazvininejad, M. Lewis, and L. Zettlemoyer, “Multilingual denoising pre-training for neural machine translation,” in Transactions of the Association for Computational Linguistics . MIT Press, 2020, pp. 726–742
2020
Earlier work this paper cites.
M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer, “Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics . ACL, 2020, pp. 7871–7880
2020
Earlier work this paper cites.
G. Ramírez-Sánchez, J. Zaragoza-Bernabeu, M. Bañón, and S. Ortiz-Rojas, “Bifixer and bicleaner: two open-source tools to clean your parallel data.” in Proceedings of the 22nd Annual Conference of the European Association for Machine Translation . European Association for Machine Translation, 2020, pp. 291–298
2020
Earlier work this paper cites.
R. Zheng, J. Chen, M. Ma, and L. Huang, “Fused acoustic and text encoding for multimodal bilingual pretraining and speech translation,” in International Conference on Machine Learning . PMLR, 2021, pp. 12 736–12 746
2021
Earlier work this paper cites.

Similar

P. Żelasko, D. Povey, J. Trmal, and S. Khudanpur, “Lhotse: a speech data representation library for the modern deep learning ecosystem,” Proceedings of NeurIPS Workshop on Data-Centric AI , 2021
2021
Cited alongside, same era.
J.-B. Alayrac et al. , “Flamingo: a visual language model for few-shot learning,” Advances in Neural Information Processing Systems , vol. 35, pp. 23 716–23 736, 2022
2022
Cited alongside, same era.
Y. Tang, H. Gong, N. Dong, C. Wang, W.-N. Hsu, J. Gu, A. Baevski, X. Li, A. Mohamed, M. Auli et al. , “Unified speech-text pre-training for speech translation and recognition,” arXiv preprint arXiv:2204.05409 , 2022
Original
2022
Cited alongside, same era.
A. Bapna, C. Cherry, Y. Zhang, Y. Jia, M. Johnson, Y. Cheng, S. Khanuja, J. Riesa, and A. Conneau, “mslam: Massively multilingual joint pre-training for speech and text,” arXiv preprint arXiv:2202.01374 , 2022
Original
2022
Cited alongside, same era.

Then

S. Mittal, O. Hrinchuk, and O. Kuchaiev, “Leveraging synthetic targets for machine translation,” 2023. [Online]. Available: https://arxiv.org/abs/2305.06155
Original
2023
Later among the works it cites.
T. Nguyen, C. V. Nguyen, V. D. Lai, H. Man, N. T. Ngo, F. Dernoncourt, R. A. Rossi, and T. H. Nguyen, “Culturax: A cleaned, enormous, and multilingual dataset for large language models in 167 languages,” 2023. [Online]. Available: https://arxiv.org/abs/2309.09400
Original
2023
Later among the works it cites.
A. Conneau, M. Ma, S. Khanuja, Y. Zhang, V. Axelrod, S. Dalmia, J. Riesa, C. Rivera, and A. Bapna, “Fleurs: Few-shot learning evaluation of universal representations of speech,” in 2022 IEEE Spoken Language Technology Workshop (SLT) . IEEE, 2023, pp. 798–805
2023
Later among the works it cites.
L. Barrault, Y.-A. Chung, M. C. Meglioli, D. Dale, N. Dong, M. Duppenthaler, P.-A. Duquenne, B. Ellis, H. Elsahar, J. Haaheim et al. , “Seamless: Multilingual expressive and streaming speech translation,” arXiv preprint arXiv:2312.05187 , 2023
Original

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

N. Goyal, C. Gao, V. Chaudhary, P.-J. Chen, G. Wenzek, D. Ju, S. Krishnan, M. Ranzato, F. Guzmán, and A. Fan, “The flores-101 evaluation benchmark for low-resource and multilingual machine translation,” Transactions of the Association for Computational Linguistics , vol. 10, pp. 522–538, 2022

2022

Cited alongside, same era.

C. Tang et al. , “Salmonn: Towards generic hearing abilities for large language models,” arXiv preprint arXiv:2310.13289 , 2023

Original

2023

Cited alongside, same era.

J. Bai et al. , “Qwen technical report,” arXiv preprint arXiv:2309.16609 , 2023

Original

2023

Cited alongside, same era.

Z. Huang, R. Ye, T. Ko, Q. Dong, S. Cheng, M. Wang, and H. Li, “Speech translation with large language models: An industrial practice,” arXiv preprint arXiv:2312.13585 , 2023

Original

2023

Cited alongside, same era.

Y. Zhang, W. Han, J. Qin, Y. Wang, A. Bapna, Z. Chen, N. Chen, B. Li, V. Axelrod, G. Wang et al. , “Google USM: Scaling automatic speech recognition beyond 100 languages,” arXiv preprint arXiv:2303.01037 , 2023

Original

2023

Cited alongside, same era.

M. Post, T. Gowda, R. Grundkiewicz, H. Khayrallah, R. Jain, and M. Junczys-Dowmunt, “SOTASTREAM: A streaming approach to machine translation training,” in Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023) , L. Tan, D. Milajevs, G. Chauhan, J. Gwinnup, and E. Rippeth, Eds. Singapore: Association for Computational Linguistics, Dec. 2023, pp. 110–119. [Online]. Available: https://aclanthology.org/2023.nlposs-1.13

2023

Cited alongside, same era.

“NVIDIA Megatron NMT en-to-any 500M.” [Online]. Available: https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/megatronnmt_en_any_500m

Cited in the paper.

“NVIDIA Megatron NMT any-to-en 500M.” [Online]. Available: https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/megatronnmt_any_en_500m

Cited in the paper.

2023

Later among the works it cites.

Z. Chen, H. Huang, A. Andrusenko, O. Hrinchuk, K. C. Puvvada, J. Li, S. Ghosh, J. Balam, and B. Ginsburg, “Salm: Speech-augmented language model with in-context learning for speech recognition and translation,” in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2024, pp. 13 521–13 525

2024

Closest in time.

Z. Kong, A. Goel, R. Badlani, W. Ping, R. Valle, and B. Catanzaro, “Audio flamingo: A novel audio language model with few-shot learning and dialogue abilities,” arXiv preprint arXiv:2402.01831 , 2024

Original

2024

Closest in time.

Z. Chen, H. Huang, O. Hrinchuk, K. C. Puvvada, N. R. Koluguri, P. Żelasko, J. Balam, and B. Ginsburg, “Bestow: Efficient and streamable speech language model with the best of two worlds in gpt and t5,” arXiv preprint arXiv:2406.19954 , 2024

Original

2024

Closest in time.

X. Xu, S. Ouyang, B. Yan, P. Fernandes, W. Chen, L. Li, G. Neubig, and S. Watanabe, “Cmu’s iwslt 2024 simultaneous speech translation system,” arXiv preprint arXiv:2408.07452 , 2024

Original

2024

Closest in time.

K. C. Puvvada, P. Żelasko, H. Huang, O. Hrinchuk, N. R. Koluguri, K. Dhawan, S. Majumdar, E. Rastorgueva, Z. Chen, V. Lavrukhin, J. Balam, and B. Ginsburg, “Less is more: Accurate speech recognition & translation without web-scale data,” in Proceedings of Interspeech , 2024

2024

Closest in time.

P. Zhang, G. Zeng, T. Wang, and W. Lu, “Tinyllama: An open-source small language model,” 2024. [Online]. Available: https://arxiv.org/abs/2401.02385

Original

2024

Closest in time.