Fetching the paper…

Reading the bibliography…

2024

Vintern-1B: An Efficient Multimodal Large Language Model for Vietnamese

Doan, Khang T., Huynh, Bao G., Hoang, Dung T. et al.

Original
PDF
Post on X

Understand

In this report, we introduce Vintern-1B, a reliable 1-billion-parameters multimodal large language model (MLLM) for Vietnamese language tasks.

By integrating the Qwen2-0.5B-Instruct language model with the InternViT-300M-448px visual model, Vintern-1B is optimized for a range of applications, including optical character recognition (OCR), document extraction, and general question-answering in Vietnamese context.
The model is fine-tuned on an extensive dataset of over 3 million image-question-answer pairs, achieving robust performance and reliable results across multiple Vietnamese language benchmarks like OpenViVQA and ViTextVQA.
Vintern-1B is small enough to fit into various on-device applications easily.

Vintern-1B: An Efficient Multimodal Large Language Model for Vietnamese · Around

Built on

N. Nguyen, T. Nguyen, V. Tran, M.-T. Tran, T. D. Ngo, T. H. Nguyen, and M. Hoai, “Dictionary-guided scene text recognition,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021, pp. 7383–7392
2021
Earlier work this paper cites.
X.-S. Vu, Q.-A. Bui, N.-V. Nguyen, T. T. H. Nguyen, and T. Vu, “Mc-ocr challenge: Mobile-captured image document recognition for vietnamese receipts,” in 2021 RIVF International Conference on Computing and Communication Technologies (RIVF) . IEEE, 2021, pp. 1–6
2021
Earlier work this paper cites.
E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” 2021. [Online]. Available: https://arxiv.org/abs/2106.09685
Original
2021
Earlier work this paper cites.
N. H. Nguyen, D. T. Vo, and K. Van Nguyen, “Uit-hwdb: Using transferring method to construct a novel benchmark for evaluating unconstrained handwriting image recognition in vietnamese,” in 2022 RIVF International Conference on Computing and Communication Technologies (RIVF) . IEEE, 2022, pp. 659–664
2022
Earlier work this paper cites.
N. H. Nguyen, D. T. Vo, K. Van Nguyen, and N. L.-T. Nguyen, “Openvivqa: Task, dataset, and multimodal fusion models for visual question answering in vietnamese,” Information Fusion , vol. 100, p. 101868, 2023
2023
Earlier work this paper cites.
J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou, “Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond,” 2023. [Online]. Available: https://arxiv.org/abs/2308.12966
Original
2023
Earlier work this paper cites.
J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023
Original
2023
Earlier work this paper cites.
“Gpt-4v(ision) system card,” 2023. [Online]. Available: https://api.semanticscholar.org/CorpusID:263218031
2023
Earlier work this paper cites.

Similar

G. Team, R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth et al. , “Gemini: a family of highly capable multimodal models,” arXiv preprint arXiv:2312.11805 , 2023
Original
2023
Cited alongside, same era.
H. Liu, C. Li, Q. Wu, and Y. J. Lee, “Visual instruction tuning,” in NeurIPS , 2023
2023
Cited alongside, same era.
H. Liu, C. Li, Y. Li, and Y. J. Lee, “Improved baselines with visual instruction tuning,” 2023
2023
Cited alongside, same era.
A. Yang, B. Yang, B. Hui, B. Zheng, B. Yu, C. Zhou, C. Li, C. Li, D. Liu, F. Huang et al. , “Qwen2 technical report,” arXiv preprint arXiv:2407.10671 , 2024
Original
2024
Cited alongside, same era.
Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, M. Zhong, Q. Zhang, X. Zhu, L. Lu et al. , “Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 24 185–24 198

Then

F. Li, R. Zhang, H. Zhang, Y. Zhang, B. Li, W. Li, Z. Ma, and C. Li, “Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models,” arXiv preprint arXiv:2407.07895 , 2024
Original
2024
Closest in time.
Y. Yao, T. Yu, A. Zhang, C. Wang, J. Cui, H. Zhu, T. Cai, H. Li, W. Zhao, Z. He et al. , “Minicpm-v: A gpt-4v level mllm on your phone,” arXiv preprint arXiv:2408.01800 , 2024
Original
2024
Closest in time.
B. Lin, Z. Tang, Y. Ye, J. Cui, B. Zhu, P. Jin, J. Zhang, M. Ning, and L. Yuan, “Moe-llava: Mixture of experts for large vision-language models,” 2024
2024
Closest in time.
H. V. Bui, H. H. Ha, P. V. Phan, and O. N. Tran, “Vistral v,” June 2024. [Online]. Available: https://huggingface.co/Vi-VLM/Vistral-V-7B
2024
Closest in time.
O. N. Tran, H. V. Bui, H. H. Ha, and P. V. Phan, “Vista,” May 2024. [Online]. Available: https://huggingface.co/datasets/Vi-VLM/Vista

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

2024

Cited alongside, same era.

M. Reid, N. Savinov, D. Teplyashin, D. Lepikhin, T. Lillicrap, J.-b. Alayrac, R. Soricut, A. Lazaridou, O. Firat, J. Schrittwieser et al. , “Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context,” arXiv preprint arXiv:2403.05530 , 2024

2024

Cited alongside, same era.

H. Liu, C. Li, Y. Li, and Y. J. Lee, “Improved baselines with visual instruction tuning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 26 296–26 306

2024

Cited alongside, same era.

2024

Closest in time.

C. Tran and H. L. Thanh, “Lavy: Vietnamese multimodal large language model,” arXiv preprint arXiv:2404.07922 , 2024

Original

2024

Closest in time.

Q. Van Nguyen, D. Q. Tran, H. Q. Pham, T. K.-B. Nguyen, N. H. Nguyen, K. Van Nguyen, and N. L.-T. Nguyen, “Vitextvqa: A large-scale visual question answering dataset for evaluating vietnamese text comprehension in images,” arXiv preprint arXiv:2404.10652 , 2024

2024

Closest in time.