Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming

Librispeech: an asr corpus based on public domain audio books

Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur · 2015

Earlier work this paper cites.

Libritts: A corpus derived from librispeech for text-to-speech

Original

Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu · 2019

Earlier work this paper cites.

wav2vec 2.0: A framework for self-supervised learning of speech representations

Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli · 2020

Earlier work this paper cites.

Mls: A large-scale multilingual dataset for speech research

Original

Vineel Pratap, Qiantong Xu, Anuroop Sriram, Gabriel Synnaeve, and Ronan Collobert · 2020

Earlier work this paper cites.

Flamingo: a visual language model for few-shot learning

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al · 2022

Earlier work this paper cites.

Beats: Audio pre-training with acoustic tokenizers

Original

Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu, Daniel Tompkins, Zhuo Chen, and Furu Wei · 2022

Earlier work this paper cites.

Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi · 2022

Earlier work this paper cites.

Lauragpt: Listen, attend, understand, and regenerate audio with gpt

Original

Qian Chen, Yunfei Chu, Zhifu Gao, Zerui Li, Kai Hu, Xiaohuan Zhou, Jin Xu, Ziyang Ma, Wen Wang, Siqi Zheng, et al · 2023

Earlier work this paper cites.

Mega-tts: Zero-shot text-to-speech at scale with intrinsic inductive bias

Original

Ziyue Jiang, Yi Ren, Zhenhui Ye, Jinglin Liu, Chen Zhang, Qian Yang, Shengpeng Ji, Rongjie Huang, Chunfeng Wang, Xiang Yin, et al · 2023

Earlier work this paper cites.

Speak, read and prompt: High-fidelity text-to-speech with minimal supervision

Eugene Kharitonov, Damien Vincent, Zalán Borsos, Raphaël Marinier, Sertan Girgin, Olivier Pietquin, Matt Sharifi, Marco Tagliasacchi, and Neil Zeghidour · 2023

Earlier work this paper cites.

Spoken question answering and speech continuation using spectrogram-powered llm

Original

Eliya Nachmani, Alon Levkovitch, Roy Hirsch, Julian Salazar, Chulayuth Asawaroengchai, Soroosh Mariooryad, Ehud Rivlin, RJ Skerry-Ryan, and Michelle Tadmor Ramanovich · 2023

Cited alongside, same era.

Robust speech recognition via large-scale weak supervision

Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever · 2023

Cited alongside, same era.

Salmonn: Towards generic hearing abilities for large language models

Original

Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang · 2023

Cited alongside, same era.

https://huggingface.co/datasets/anthropic/hh-rlhf, 2024

Anthropic · 2024

Cited alongside, same era.

Qwen2-audio technical report

Original

Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, et al · 2024

Voicebox: Text-guided multilingual universal speech generation at scale

Matthew Le, Apoorv Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar, et al · 2024

Closest in time.

Improved baselines with visual instruction tuning

Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee · 2024

Closest in time.

llama3.1, 2024

meta · 2024

Closest in time.

https://mistral.ai/, 2024

mixtral · 2024

Closest in time.

https://openai.com/, 2024

openai · 2024

Closest in time.

https://huggingface.co/datasets/openassistant/oasst1, 2024

OpenAssistan · 2024

Closest in time.

https://huggingface.co/datasets/sayan1101/identity-finetune-data, 2024