2024

xGen-MM-Vid (BLIP-3-Video): You Only Need 32 Tokens to Represent a Video Even in VLMs

Ryoo, Michael S., Zhou, Honglu, Kendre, Shrikant et al.

Understand

We present xGen-MM-Vid (BLIP-3-Video): a multimodal language model for videos, particularly designed to efficiently capture temporal information over multiple frames.

BLIP-3-Video takes advantage of the 'temporal encoder' in addition to the conventional visual tokenizer, which maps a sequence of tokens over multiple frames into a compact set of visual tokens.
This enables BLIP3-Video to use much fewer visual tokens than its competing models (e.g., 32 vs.
4608 tokens).

Built on

Tgif-qa: Toward spatio-temporal reasoning in visual question answering
Jang, Y., Song, Y., Yu, Y., Kim, Y., and Kim, G · 2017
Earlier work this paper cites.
Video question answering via gradually refined attention over appearance and motion
Xu, D., Zhao, Z., Xiao, J., Wu, F., Zhang, H., He, X., and Zhuang, Y · 2017
Earlier work this paper cites.
Activitynet-qa: A dataset for understanding complex web videos via question answering
Yu, Z., Xu, D., Yu, J., Yu, T., Zhao, Z., Zhuang, Y., and Tao, D · 2019
Earlier work this paper cites.
Learning transferable visual models from natural language supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I · 2021
Earlier work this paper cites.
Tokenlearner: Adaptive space-time tokenization for videos
Ryoo, M., Piergiovanni, A., Arnab, A., Dehghani, M., and Angelova, A · 2021
Earlier work this paper cites.
Next-qa: Next phase of question-answering to explaining temporal actions
Xiao, J., Shang, X., Yao, A., and Chua, T.-S · 2021
Earlier work this paper cites.
Flamingo: a visual language model for few-shot learning
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., Ring, R., Rutherford, E., Cabi, S., Han, T., Gong, Z., Samangooei, S., Monteiro, M., Menick, J., Borgeaud, S., Brock, A., Nematzadeh, A., Sharifzadeh, S., Binkowski, M., Barreira, R., Vinyals, O., Zisserman, A., and Simonyan, K · 2022
Earlier work this paper cites.
Token merging: Your vit but faster
Original
Bolya, D., Fu, C.-Y., Dai, X., Zhang, P., Feichtenhofer, C., and Hoffman, J · 2022
Earlier work this paper cites.
Perceiver IO: A general architecture for structured inputs & outputs
Jaegle, A., Borgeaud, S., Alayrac, J., Doersch, C., Ionescu, C., Ding, D., Koppula, S., Zoran, D., Brock, A., Shelhamer, E., Hénaff, O. J., Botvinick, M. M., Zisserman, A., Vinyals, O., and Carreira, J · 2022
Earlier work this paper cites.
Grit: A generative region-to-text transformer for object understanding
Original
Wu, J., Wang, J., Yang, Z., Gan, Z., Liu, Z., Yuan, J., and Wang, L · 2022
Earlier work this paper cites.
Coca: Contrastive captioners are image-text foundation models
Original
Yu, J., Wang, Z., Vasudevan, V., Yeung, L., Seyedhosseini, M., and Wu, Y · 2022
Earlier work this paper cites.
Qwen technical report, 2023
Original
Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., Hui, B., Ji, L., Li, M., Lin, J., Lin, R., Liu, D., Liu, G., Lu, C., Lu, K., Ma, J., Men, R., Ren, X., Ren, X., Tan, C., Tan, S., Tu, J., Wang, P., Wang, S., Wang, W., Wu, S., Xu, B., Xu, J., Yang, A., Yang, H., Yang, J., Yang, S., Yao, Y., Yu, B., Yuan, H., Yuan, Z., Zhang, J., Zhang, X., Zhang, Y., Zhang, Z., Zhou, C., Zhou, J., Zhou, X., and Zhu, T · 2023
Earlier work this paper cites.
Instructblip: Towards general-purpose vision-language models with instruction tuning
Dai, W., Li, J., Li, D., Tiong, A. M. H., Zhao, J., Wang, W., Li, B., Fung, P., and Hoi, S. C. H · 2023
Earlier work this paper cites.
OBELICS: an open web-scale filtered dataset of interleaved image-text documents
Laurençon, H., Saulnier, L., Tronchon, L., Bekman, S., Singh, A., Lozhkov, A., Wang, T., Karamcheti, S., Rush, A. M., Kiela, D., Cord, M., and Sanh, V · 2023
Earlier work this paper cites.
Video-llava: Learning united visual representation by alignment before projection
Original
Lin, B., Zhu, B., Ye, Y., Ning, M., Jin, P., and Yuan, L · 2023
Earlier work this paper cites.

Similar

Visual instruction tuning, 2023
Liu, H., Li, C., Wu, Q., and Lee, Y. J · 2023
Cited alongside, same era.
Testa: Temporal-spatial token aggregation for long-form video-language understanding
Original
Ren, S., Chen, S., Li, S., Sun, X., and Hou, L · 2023
Cited alongside, same era.
Token turing machines
Ryoo, M. S., Gopalakrishnan, K., Kahatapitiya, K., Xiao, T., Rao, K., Stone, A., Lu, Y., Ibarz, J., and Arnab, A · 2023
Cited alongside, same era.
Sigmoid loss for language image pre-training
Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L · 2023
Cited alongside, same era.
Video-llama: An instruction-tuned audio-visual language model for video understanding
Original
Zhang, H., Li, X., and Bing, L · 2023
Cited alongside, same era.
Phi-3 technical report: A highly capable language model locally on your phone, 2024
Original
Abdin, M., Jacobs, S. A., Awan, A. A., Aneja, J., Awadallah, A., Awadalla, H., Bach, N., Bahree, A., Bakhtiari, A., Bao, J., Behl, H., Benhaim, A., Bilenko, M., Bjorck, J., Bubeck, S., Cai, Q., Cai, M., Mendes, C. C. T., Chen, W., Chaudhary, V., Chen, D., Chen, D., Chen, Y.-C., Chen, Y.-L., Chopra, P., Dai, X., Giorno, A. D., de Rosa, G., Dixon, M., Eldan, R., Fragoso, V., Iter, D., Gao, M., Gao, M., Gao, J., Garg, A., Goswami, A., Gunasekar, S., Haider, E., Hao, J., Hewett, R. J., Huynh, J., Javaheripi, M., Jin, X., Kauffmann, P., Karampatziakis, N., Kim, D., Khademi, M., Kurilenko, L., Lee, J. R., Lee, Y. T., Li, Y., Li, Y., Liang, C., Liden, L., Liu, C., Liu, M., Liu, W., Lin, E., Lin, Z., Luo, C., Madan, P., Mazzola, M., Mitra, A., Modi, H., Nguyen, A., Norick, B., Patra, B., Perez-Becker, D., Portet, T., Pryzant, R., Qin, H., Radmilac, M., Rosset, C., Roy, S., Ruwase, O., Saarikivi, O., Saied, A., Salim, A., Santacroce, M., Shah, S., Shang, N., Sharma, H., Shukla, S., Song, X., Tanaka, M., Tupini, A., Wang, X., Wang, L., Wang, C., Wang, Y., Ward, R., Wang, G., Witte, P., Wu, H., Wyatt, M., Xiao, B., Xu, C., Xu, J., Xu, W., Yadav, S., Yang, F., Yang, J., Yang, Z., Yang, Y., Yu, D., Yuan, L., Zhang, C., Zhang, C., Zhang, J., Zhang, L. L., Zhang, Y., Zhang, Y., Zhang, Y., and Zhou, X · 2024
Cited alongside, same era.
Minigpt4-video: Advancing multimodal llms for video understanding with interleaved visual-textual tokens
Original
Ataallah, K., Shen, X., Abdelrahman, E., Sleiman, E., Zhu, D., Ding, J., and Elhoseiny, M · 2024
Cited alongside, same era.
Mint-1t: Scaling open-source multimodal data by 10x: A multimodal dataset with one trillion tokens, 2024
Original
Awadalla, A., Xue, L., Lo, O., Shu, M., Lee, H., Guha, E. K., Jordan, M., Shen, S., Awadalla, M., Savarese, S., Xiong, C., Xu, R., Choi, Y., and Schmidt, L · 2024
Cited alongside, same era.
An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models
Original
Chen, L., Zhao, H., Liu, T., Bai, S., Lin, J., Zhou, C., and Chang, B · 2024
Cited alongside, same era.
Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models, 2024
Original
Deitke, M., Clark, C., Lee, S., Tripathi, R., Yang, Y., Park, J. S., Salehi, M., Muennighoff, N., Lo, K., Soldaini, L., Lu, J., Anderson, T., Bransom, E., Ehsani, K., Ngo, H., Chen, Y., Patel, A., Yatskar, M., Callison-Burch, C., Head, A., Hendrix, R., Bastani, F., VanderBilt, E., Lambert, N., Chou, Y., Chheda, A., Sparks, J., Skjonsberg, S., Schmitz, M., Sarnat, A., Bischoff, B., Walsh, P., Newell, C., Wolters, P., Gupta, T., Zeng, K.-H., Borchardt, J., Groeneveld, D., Dumas, J., Nam, C., Lebrecht, S., Wittlif, C., Schoenick, C., Michel, O., Krishna, R., Weihs, L., Smith, N. A., Hajishirzi, H., Girshick, R., Farhadi, A., and Kembhavi, A · 2024
Cited alongside, same era.
The llama 3 herd of models, 2024
Original
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., Goyal, A., Hartshorn, A., Yang, A., Mitra, A., Sravankumar, A., et al · 2024
Cited alongside, same era.
Lazyllm: Dynamic token pruning for efficient long context llm inference
Original
Fu, Q., Cho, M., Merth, T., Mehta, S., Rastegari, M., and Najibi, M · 2024
Cited alongside, same era.

Then

Chat-univi: Unified visual representation empowers large language models with image and video understanding
Jin, P., Takanobu, R., Zhang, W., Cao, X., and Yuan, L · 2024
Closest in time.
Miradata: A large-scale video dataset with long durations and structured captions, 2024
Original
Ju, X., Gao, Y., Zhang, Z., Yuan, Z., Wang, X., Zeng, A., Xiong, Y., Xu, Q., and Shan, Y · 2024
Closest in time.
Language repository for long video understanding, 2024
Original
Kahatapitiya, K., Ranasinghe, K., Park, J., and Ryoo, M. S · 2024
Closest in time.
An image grid can be worth a video: Zero-shot video question answering using a vlm, 2024
Original
Kim, W., Choi, C., Lee, W., and Rhee, W · 2024
Closest in time.
Building and better understanding vision-language models: insights and future directions., 2024
Laurençon, H., Marafioti, A., Sanh, V., and Tronchon, L · 2024
Closest in time.
Llava-next: Tackling multi-image, video, and 3d in large multimodal models, June 2024b
Li, F., Zhang, R., Zhang, H., Zhang, Y., Li, B., Li, W., Ma, Z., and Li, C · 2024
Closest in time.
Kangaroo: A powerful video-language model supporting long-context video input, 2024
Original
Liu, J., Wang, Y., Ma, H., Wu, X., Ma, X., Wei, X., Jiao, J., Wu, E., and Hu, J · 2024
Closest in time.
Video-chatgpt: Towards detailed video understanding via large vision and language models
Maaz, M., Rasheed, H., Khan, S., and Khan, F. S · 2024
Closest in time.
Too many frames, not all useful: Efficient strategies for long-form video qa
Original
Park, J. S., Ranasinghe, K., Kahatapitiya, K., Ryoo, W., Kim, D., and Ryoo, M. S · 2024
Closest in time.
Mirasol3b: A multimodal autoregressive model for time-aligned and contextual modalities
Piergiovanni, A., Noble, I., Kim, D., Ryoo, M. S., Gomes, V., and Angelova, A · 2024
Closest in time.
Direct preference optimization: Your language model is secretly a reward model
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C · 2024
Closest in time.
Llava-prumerge: Adaptive token reduction for efficient large multimodal models
Original
Shang, Y., Cai, M., Xu, B., Lee, Y. J., and Yan, Y · 2024
Closest in time.
Look-m: Look-once optimization in kv cache for efficient multimodal long-context inference
Original
Wan, Z., Wu, Z., Liu, C., Huang, J., Zhu, Z., Jin, P., Wang, L., and Yuan, L · 2024
Closest in time.
xgen-mm (blip-3): A family of open large multimodal models
Original
Xue, L., Shu, M., Awadalla, A., Wang, J., Yan, A., Purushwalkam, S., Zhou, H., Prabhu, V., Dai, Y., Ryoo, M. S., et al · 2024
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…