Fetching the paper…
Reading the bibliography…
Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber · 2006
Earlier work this paper cites.
Librispeech: an asr corpus based on public domain audio books
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur · 2015
Earlier work this paper cites.
Aishell-1: An open-source mandarin speech corpus and a speech recognition baseline
Hui Bu, Jiayu Du, Xingyu Na, Bengu Wu, and Hao Zheng · 2017
Earlier work this paper cites.
Language models are few-shot learners
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared~D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et~al · 2020
Earlier work this paper cites.
wav2vec 2.0: A framework for self-supervised learning of speech representations
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli · 2020
Earlier work this paper cites.
Geometry-guided dense perspective network for speech-driven facial animation
Jingying Liu, Binyuan Hui, Kun Li, Yunke Liu, Yu-Kun Lai, Yuxiang Zhang, Yebin Liu, and Jingyu Yang · 2021
Earlier work this paper cites.
Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut · 2021
Earlier work this paper cites.
Ai2d-rst: A multimodal corpus of 1000 primary school science diagrams
Tuomo Hiippala, Malihe Alikhani, Jonas Haverinen, Timo Kalliokoski, Evanfiya Logacheva, Serafina Orekhova, Aino Tuomainen, Matthew Stone, and John~A Bateman · 2021
Earlier work this paper cites.
Gigaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audio
Guoguo Chen, Shuzhou Chai, Guanbo Wang, Jiayu Du, Wei-Qiang Zhang, Chao Weng, Dan Su, Daniel Povey, Jan Trmal, Junbo Zhang, et~al · 2021
Earlier work this paper cites.
Improving multimodal speech recognition by data augmentation and speech representations
Dan Oneată and Horia Cucu · 2022
Earlier work this paper cites.
Learn to explain: Multimodal reasoning via thought chains for science question answering
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan · 2022
Earlier work this paper cites.
Wukong: A 100 million large-scale chinese cross-modal pre-training benchmark
Jiaxi Gu, Xiaojun Meng, Guansong Lu, Lu~Hou, Niu Minzhe, Xiaodan Liang, Lewei Yao, Runhui Huang, Wei Zhang, Xin Jiang, et~al · 2022
Earlier work this paper cites.
Laion-5b: An open large-scale dataset for training next generation image-text models
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et~al · 2022
Earlier work this paper cites.
Wenetspeech: A 10000+ hours multi-domain mandarin corpus for speech recognition
Binbin Zhang, Hang Lv, Pengcheng Guo, Qijie Shao, Chao Yang, Lei Xie, Xin Xu, Hui Bu, Xiaoyu Chen, Chenchen Zeng, et~al · 2022
Earlier work this paper cites.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong~Jae Lee · 2023
Earlier work this paper cites.
Video-llama: An instruction-tuned audio-visual language model for video understanding
Hang Zhang, Xin Li, and Lidong Bing · 2023
Earlier work this paper cites.
Audio-visual llm for video understanding
Fangxun Shu, Lei Zhang, Hao Jiang, and Cihang Xie · 2023
Earlier work this paper cites.
Speech-to-text and text-to-speech recognition using deep learning
V~Madhusudhana Reddy, T~Vaishnavi, and K~Pavan Kumar · 2023
Earlier work this paper cites.
Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities
Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu · 2023
Earlier work this paper cites.
Gpt-4 technical report
OpenAI · 2023
Earlier work this paper cites.
Llama: Open and efficient foundation language models
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et~al · 2023
Earlier work this paper cites.
Llama 2: Open foundation and fine-tuned chat models
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et~al · 2023
Earlier work this paper cites.
Stanford alpaca: An instruction-following llama model, 2023
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori~B Hashimoto · 2023
Earlier work this paper cites.
Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality
Wei-Lin Chiang, Zhuohan Li, Zi~Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph~E Gonzalez, et~al · 2023
Earlier work this paper cites.
Albert~Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra~Singh Chaplot, Diego de~las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et~al · 2023
Earlier work this paper cites.
Qwen-vl: A frontier large vision-language model with versatile abilities
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou · 2023
Earlier work this paper cites.
Video-llava: Learning united visual representation by alignment before projection
Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li~Yuan · 2023
Cited alongside, same era.
E-chat: Emotion-sensitive spoken dialogue system with large language models
Hongfei Xue, Yuhao Liang, Bingshen Mu, Shiliang Zhang, Qian Chen, and Lei Xie · 2023
Cited alongside, same era.
Paralinguistics-enhanced large language modeling of spoken dialogue
Guan-Ting Lin, Prashanth~Gurunath Shivakumar, Ankur Gandhe, Chao-Han~Huck Yang, Yile Gu, Shalini Ghosh, Andreas Stolcke, Hung-yi Lee, and Ivan Bulyko · 2023
Cited alongside, same era.
Hello gpt-4o
OpenAI · 2023
Cited alongside, same era.
Long context transfer from language to vision
Peiyuan Zhang, Kaichen Zhang, Bo~Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu · 2024
Later among the works it cites.
https://openai.com/index/hello-gpt-4o/ , 2024
OpenAI · 2024
Later among the works it cites.
Mini-omni2: Towards open-source gpt-4o model with vision, speech and duplex
Zhifei Xie and Changqiao Wu · 2024
Later among the works it cites.
Llama-omni: Seamless speech interaction with large language models
Qingkai Fang, Shoutao Guo, Yan Zhou, Zhengrui Ma, Shaolei Zhang, and Yang Feng · 2024
Later among the works it cites.
Moshi: a speech-text foundation model for real-time dialogue
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew~M Dai, Anja Hauth, et~al · 2023
Cited alongside, same era.
Sharegpt4v: Improving large multi-modal models with better captions
Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin · 2023
Cited alongside, same era.
To see is to believe: Prompting gpt-4v for better visual instruction tuning
Junke Wang, Lingchen Meng, Zejia Weng, Bo~He, Zuxuan Wu, and Yu-Gang Jiang · 2023
Cited alongside, same era.
Anytext: Multilingual visual text generation and editing
Yuxiang Tuo, Wangmeng Xiang, Jun-Yan He, Yifeng Geng, and Xuansong Xie · 2023
Cited alongside, same era.
Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Guohai Xu, Chenliang Li, Junfeng Tian, Qi~Qian, Ji~Zhang, et~al · 2023
Cited alongside, same era.
Vila: On pre-training for visual language models, 2023
Ji~Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han · 2023
Cited alongside, same era.
Pan Zhang, Xiaoyi Dong~Bin Wang, Yuhang Cao, Chao Xu, Linke Ouyang, Zhiyuan Zhao, Shuangrui Ding, Songyang Zhang, Haodong Duan, Hang Yan, et~al · 2023
Cited alongside, same era.
Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu~Qiao, and Jifeng Dai · 2023
Cited alongside, same era.
Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour · 2024
Later among the works it cites.
How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites
Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et~al · 2024
Later among the works it cites.
Freeze-omni: A smart and low latency speech-to-speech dialogue model with frozen llm
Xiong Wang, Yangze Li, Chaoyou Fu, Lei Xie, Ke~Li, Xing Sun, and Long Ma · 2024
Later among the works it cites.
Fewer-token neural speech codec with time-invariant codes
Yong Ren, Tao Wang, Jiangyan Yi, Le~Xu, Jianhua Tao, Chu~Yuan Zhang, and Junzuo Zhou · 2024
Later among the works it cites.
Allava: Harnessing gpt4v-synthesized data for a lite vision-language model, 2024
Guiming~Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, and Benyou Wang · 2024
Later among the works it cites.
Chatqa: Surpassing gpt-4 on conversational qa and rag
Zihan Liu, Wei Ping, Rajarshi Roy, Peng Xu, Chankyu Lee, Mohammad Shoeybi, and Bryan Catanzaro · 2024
Later among the works it cites.
Llava-onevision: Easy visual task transfer
Bo~Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et~al · 2024
Later among the works it cites.
Sharegemini: Scaling up video caption data for multimodal large language models, June 2024
Share · 2024
Later among the works it cites.
Llava-next: Stronger llms supercharge multimodal capabilities in the wild, May 2024
Bo~Li, Kaichen Zhang, Hao Zhang, Dong Guo, Renrui Zhang, Feng Li, Yuanhan Zhang, Ziwei Liu, and Chunyuan Li · 2024
Later among the works it cites.
Cogvlm2: Visual language models for image and video understanding
Wenyi Hong, Weihan Wang, Ming Ding, Wenmeng Yu, Qingsong Lv, Yan Wang, Yean Cheng, Shiyu Huang, Junhui Ji, Zhao Xue, et~al · 2024
Later among the works it cites.
Ovis: Structural embedding alignment for multimodal large language model
Shiyin Lu, Yang Li, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Han-Jia Ye · 2024
Later among the works it cites.
Are we on the right way for evaluating large vision-language models?
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu~Qiao, Dahua Lin, et~al · 2024
Later among the works it cites.
Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge~Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et~al · 2024
Later among the works it cites.
Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models
Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et~al · 2024
Later among the works it cites.
Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis
Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et~al · 2024
Later among the works it cites.
Mvbench: A comprehensive multi-modal video understanding benchmark
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi~Wang, Yi~Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et~al · 2024
Later among the works it cites.
Tempcompass: Do video llms really understand videos?
Yuanxin Liu, Shicheng Li, Yi~Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu~Sun, and Lu~Hou · 2024
Later among the works it cites.
Mini-omni2: Towards open-source gpt-4o with vision, speech and duplex capabilities
Zhifei Xie and Changqiao Wu · 2024
Later among the works it cites.
Speechact: Towards generating whole-body motion from speech
Jinsong Zhang, Minjie Zhu, Yuxiang Zhang, Zerong Zheng, Yebin Liu, and Kun Li · 2025
Closest in time.
Dycrowd: Towards dynamic crowd reconstruction from a large-scene video
Hao Wen, Hongbo Kang, Jian Ma, Jing Huang, Yuanwang Yang, Haozhe Lin, Yu-Kun Lai, and Kun Li · 2025
Closest in time.
Empowering llms with pseudo-untrimmed videos for audio-visual temporal understanding
Yunlong Tang, Daiki Shimada, Jing Bi, Mingqian Feng, Hang Hua, and Chenliang Xu · 2025
Closest in time.
Logavatar: Local gaussian splatting for human avatar modeling from monocular video
Jinsong Zhang, Xiongzheng Li, Hailong Jia, Jin Li, Zhuo Su, Guidong Wang, and Kun Li · 2025
Closest in time.