Long-VITA: Scaling Large Multi-modal Models to 1 Million Tokens with Leading Short-Context Accuracy

A. Kembhavi, M. Salvato, E. Kolve, M. Seo, H. Hajishirzi, and A. Farhadi, “a Diagram Is Worth a Dozen Images,”

2016

Earlier work this paper cites.

Q. Huang, Y. Xiong, A. Rao, J. Wang, and D. Lin, “MovieNet: A Holistic Dataset for Movie Understanding,”

2020

Earlier work this paper cites.

E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-Rank Adaptation of Large Language Models,” 2021. [Online]. Available:

Original

2021

Earlier work this paper cites.

J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, R. Ring, E. Rutherford, S. Cabi, T. Han, Z. Gong, S. Samangooei, M. Monteiro, J. Menick, S. Borgeaud, A. Brock, A. Nematzadeh, S. Sharifzadeh, M. Binkowski, R. Barreira, O. Vinyals, A. Zisserman, and K. Simonyan, “Flamingo: A Visual Language Model for Few-Shot Learning,” 2022

2022

Earlier work this paper cites.

OpenAI, “GPT-4,” 2023. [Online]. Available:

2023

Earlier work this paper cites.

H. Liu, C. Li, Y. Li, and Y. J. Lee, “Improved Baselines with Visual Instruction Tuning,” 2023. [Online]. Available:

Original

2023

Earlier work this paper cites.

J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou, “Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond,” 2023. [Online]. Available:

Original

2023

Earlier work this paper cites.

J. Li, D. Li, S. Savarese, and S. Hoi, “BLIP-2: Bootstrapping Language-Image Pre-Training with Frozen Image Encoders and Large Language Models,” 2023. [Online]. Available:

Original

2023

Earlier work this paper cites.

H. Liu, C. Li, Q. Wu, and Y. J. Lee, “Visual Instruction Tuning,” 2023. [Online]. Available:

Original

2023

Earlier work this paper cites.

Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, M. Zhong, Q. Zhang, X. Zhu, L. Lu, B. Li, P. Luo, T. Lu, Y. Qiao, and J. Dai, “InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks,” 2023. [Online]. Available:

Original

2023

Earlier work this paper cites.

Y. Liu, H. Duan, Y. Zhang, B. Li, S. Zhang, W. Zhao, Y. Yuan, J. Wang, C. He, Z. Liu, K. Chen, and D. Lin, “MMBench: Is Your Multi-Modal Model an All-Around Player?” 2023. [Online]. Available:

Original

2023

Earlier work this paper cites.

C. Fu, P. Chen, Y. Shen, Y. Qin, M. Zhang, X. Lin, J. Yang, X. Zheng, K. Li, X. Sun, Y. Wu, and R. Ji, “MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models,” 2023. [Online]. Available:

Original

2023

Earlier work this paper cites.

S. Chen, S. Wong, L. Chen, and Y. Tian, “Extending Context Window of Large Language Models Via Positional Interpolation,” 2023. [Online]. Available:

Original

2023

Earlier work this paper cites.

B. Peng, J. Quesnelle, H. Fan, E. Shippole, and N. Research, “YaRN: Efficient Context Window Extension of Large Language Models,” 2023. [Online]. Available:

Original

2023

Earlier work this paper cites.

Y. Chen, S. Qian, H. Tang, X. Lai, Z. Liu, S. Han, and J. Jia, “LongLoRA: Efficient Fine-Tuning of Long-Context Large Language Models,” 2023. [Online]. Available:

Original

2023

Earlier work this paper cites.

J. Wang, L. Meng, Z. Weng, B. He, Z. Wu, and Y.-G. Jiang, “to See Is to Believe: Prompting GPT-4V for Better Visual Instruction Tuning,” 2023. [Online]. Available:

Original

2023

Earlier work this paper cites.

L. Chen, J. Li, . §2, X. Dong, P. Zhang, C. He, J. Wang, F. Zhao, and D. Lin, “ShareGPT4V: Improving Large Multi-Modal Models with Better Captions,” 2023. [Online]. Available:

Original

2023

Earlier work this paper cites.

Teknium, “OpenHermes 2.5: An Open Dataset of Synthetic Data for Generalist LLM Assistants,” 2023. [Online]. Available:

2023

Earlier work this paper cites.

C. Zhou, P. Liu, P. Xu, S. Iyer, J. Sun, Y. Mao, X. Ma, A. Efrat, P. Yu, L. Yu, S. Zhang, G. Ghosh, M. Lewis, L. Zettlemoyer, and O. Levy, “LIMA: Less Is More for Alignment,” 2023. [Online]. Available:

Original

2023

Earlier work this paper cites.

M. Conover, M. Hayes, A. Mathur, J. Xie, J. Wan, S. Shah, A. Ghodsi, P. Wendell, M. Zaharia, and R. Xin, “Free Dolly: Introducing the World’s First Truly Open Instruction-Tuned LLM,” 2023. [Online]. Available:

2023

Earlier work this paper cites.

A. Unified, “Atlas math sets,” 2023. [Online]. Available:

2023

Earlier work this paper cites.

L. Tiedong, “Goat,” 2023. [Online]. Available:

2023

Earlier work this paper cites.

G. Li, H. A. A. K. Hammoud, H. Itani, D. Khizbullin, and B. Ghanem, “CAMEL: Communicative Agents for ”Mind” Exploration of Large Language Model Society,” 2023. [Online]. Available:

Original

2023

Earlier work this paper cites.

Y. Yu, “”Paraphrasing the Original Text” Makes High Accuracy Long-Context QA,” 2023. [Online]. Available:

Original

2023

Earlier work this paper cites.

A. Köksal, T. Schick, A. Korhonen, and H. Schütze, “LongForm: Optimizing Instruction Tuning for Long Text Generation with Corpus Extraction,” 2023. [Online]. Available:

Original

2023

Earlier work this paper cites.

J. Yang, “LongQLoRA: Efficient and Effective Method to Extend Context Length of Large Language Models,” 2023. [Online]. Available:

Original

2023

Earlier work this paper cites.

T. Computer, “Long Data Collections,” 2023. [Online]. Available:

2023

Cited alongside, same era.

T. Guan, F. Liu, X. Wu, R. Xian, Z. Li, X. Liu, X. Wang, L. Chen, F. Huang, Y. Yacoob, D. Manocha, and T. Zhou, “HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models,” 2023. [Online]. Available:

Original

2023

Cited alongside, same era.

Y. Liu, Z. Li, B. Yang, C. Li, X.-C. Yin, C.-L. Liu, L. Jin, and X. Bai, “on the Hidden Mystery of OCR in Large Multimodal Models,” 2023. [Online]. Available:

Original

2023

Cited alongside, same era.

J. Lin, H. Yin, W. Ping, Y. Lu, P. Molchanov, A. Tao, H. Mao, J. Kautz, M. Shoeybi, and S. Han, “VILA: On Pre-Training for Visual Language Models,” 2023. [Online]. Available:

Original

2023

Cited alongside, same era.

X. Yue, Y. Ni, K. Zhang, T. Zheng, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Y. Sun, C. Wei, B. Yu, R. Yuan, R. Sun, M. Yin, B. Zheng, Z. Yang, Y. Liu, W. Huang, H. Sun, Y. Su, and W. Chen, “MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI,” 2023. [Online]. Available:

M. Maaz, H. Rasheed, S. Khan, and F. S. Khan, “VideoGPT+: Integrating Image and Video Encoders for Enhanced Video Understanding,” 2024. [Online]. Available:

Original

2024

Later among the works it cites.

“ShareGemini: Scaling up Video Caption Data for Multimodal Large Language Models,” 2024. [Online]. Available:

2024

Later among the works it cites.

L. Yu, W. Jiang, H. Shi, J. Yu, Z. Liu, Y. Zhang, J. T. Kwok, Z. Li, A. Weller, and W. Liu, “Metamath: Bootstrap Your Own Mathematical Questions for Large Language Models,”

2024

Later among the works it cites.

X. Yue, X. Qu, G. Zhang, Y. Fu, W. Huang, H. Sun, Y. Su, and W. Chen, “Mammoth: Building Math Generalist Models Through Hybrid Instruction Tuning,”

2024

Later among the works it cites.

A. Mitra, H. Khanpour, C. Rosset, and A. Awadallah, “Orca-Math: Unlocking the Potential of SLMs in Grade School Math,” 2024. [Online]. Available: