Long Video Understanding with Learnable Retrieval in Video-Language Models

Around Connect

Reading the bibliography…

Fetching the paper…

Built on

R. F. Thompson and S. A. Madigan, Memory: the key to consciousness . Princeton University Press, 2013, vol. 3
2013
Earlier work this paper cites.
F. Caba Heilbron, V. Escorcia, B. Ghanem, and J. Carlos Niebles, “Activitynet: A large-scale video benchmark for human activity understanding,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2015, pp. 961–970
2015
Earlier work this paper cites.
Z. Yuan, S. Sun, L. Duan, C. Li, X. Wu, and C. Xu, “Adversarial multimodal network for movie story question answering,” IEEE Transactions on Multimedia , vol. 23, pp. 1744–1756, 2020
2020
Earlier work this paper cites.
W. Zhang, S. Tang, Y. Cao, S. Pu, F. Wu, and Y. Zhuang, “Frame augmented alternating attention network for video question answering,” IEEE Transactions on Multimedia , vol. 22, no. 4, pp. 1032–1041, 2020
2020
Earlier work this paper cites.
S. Castro, M. Azab, J. Stroud, C. Noujaim, R. Wang, J. Deng, and R. Mihalcea, “LifeQA: A real-life dataset for video question answering,” in Proceedings of the Twelfth Language Resources and Evaluation Conference , 2020, pp. 4352–4358
2020
Earlier work this paper cites.
J. Wang, B.-K. Bao, and C. Xu, “Dualvgr: A dual-visual graph reasoning unit for video question answering,” IEEE Transactions on Multimedia , vol. 24, pp. 3369–3380, 2021
2021
Earlier work this paper cites.
Z. Guo, J. Zhao, L. Jiao, X. Liu, and F. Liu, “A universal quaternion hypergraph network for multimodal video question answering,” IEEE Transactions on Multimedia , vol. 25, pp. 38–49, 2021
2021
Earlier work this paper cites.
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al. , “Learning transferable visual models from natural language supervision,” in International conference on machine learning . PMLR, 2021, pp. 8748–8763
2021
Earlier work this paper cites.
J. Xiao, X. Shang, A. Yao, and T.-S. Chua, “Next-qa: Next phase of question-answering to explaining temporal actions,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2021, pp. 9777–9786
2021
Earlier work this paper cites.
A. Yang, A. Miech, J. Sivic, I. Laptev, and C. Schmid, “Just ask: Learning to answer questions from millions of narrated videos,” in Proceedings of the IEEE/CVF international conference on computer vision , 2021, pp. 1686–1697
2021
Earlier work this paper cites.
J. Jiang, Z. Liu, and N. Zheng, “Livlr: A lightweight visual-linguistic reasoning framework for video question answering,” IEEE Transactions on Multimedia , vol. 25, pp. 5002–5013, 2022
2022
Earlier work this paper cites.
J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds et al. , “Flamingo: a visual language model for few-shot learning,” Advances in Neural Information Processing Systems , vol. 35, pp. 23 716–23 736, 2022
2022
Earlier work this paper cites.
S. Castro, N. Deng, P. Huang, M. G. Burzo, and R. Mihalcea, “In-the-wild video question answering,” in International Conference on Computational Linguistics , 2022, pp. 5613–5635
2022
Earlier work this paper cites.
L. Bärmann and A. Waibel, “Where did i leave my keys? - episodic-memory-based question answering on egocentric videos,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops , June 2022, pp. 1560–1568
2022
Earlier work this paper cites.
A. Yang, A. Miech, J. Sivic, I. Laptev, and C. Schmid, “Zero-shot video question answering via frozen bidirectional language models,” Advances in Neural Information Processing Systems , vol. 35, pp. 124–141, 2022
2022
Earlier work this paper cites.
Y. Wang, K. Li, Y. Li, Y. He, B. Huang, Z. Zhao, H. Zhang, J. Xu, Y. Liu, Z. Wang et al. , “Internvideo: General video foundation models via generative and discriminative learning,” arXiv preprint arXiv:2212.03191 , 2022
Original
2022
Earlier work this paper cites.
J. Xiao, A. Yao, Z. Liu, Y. Li, W. Ji, and T.-S. Chua, “Video as conditional graph hierarchy for multi-granular question answering,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 36, no. 3, 2022, pp. 2804–2812
2022
Earlier work this paper cites.
J. Xiao, P. Zhou, T.-S. Chua, and S. Yan, “Video graph transformer for video question answering,” 2022. [Online]. Available: https://arxiv.org/abs/2207.05342
Original
2022
Earlier work this paper cites.
L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray et al. , “Training language models to follow instructions with human feedback,” Advances in Neural Information Processing Systems , vol. 35, pp. 27 730–27 744, 2022
2022
Earlier work this paper cites.
M. Maaz, H. Rasheed, S. Khan, and F. S. Khan, “Video-ChatGPT: Towards detailed video understanding via large vision and language models,” arXiv preprint arXiv:2306.05424 , 2023
Original
2023
Earlier work this paper cites.
J. Wang, D. Chen, C. Luo, X. Dai, L. Yuan, Z. Wu, and Y.-G. Jiang, “Chatvideo: A tracklet-centric multimodal and versatile video understanding system,” arXiv preprint arXiv:2304.14407 , 2023
Original
2023
Earlier work this paper cites.

Similar

K. Li, Y. He, Y. Wang, Y. Li, W. Wang, P. Luo, Y. Wang, L. Wang, and Y. Qiao, “Videochat: Chat-centric video understanding,” arXiv preprint arXiv:2305.06355 , 2023
Original
2023
Cited alongside, same era.
H. Zhang, X. Li, and L. Bing, “Video-LLaMA: An instruction-tuned audio-visual language model for video understanding,” arXiv preprint arXiv:2306.02858 , 2023
Original
2023
Cited alongside, same era.
T. Qian, R. Cui, J. Chen, P. Peng, X. Guo, and Y.-G. Jiang, “Locate before answering: Answer guided question localization for video question answering,” IEEE Transactions on Multimedia , 2023
2023
Cited alongside, same era.
F. Zhang, R. Wang, F. Zhou, Y. Luo, and J. Li, “Psam: Parameter-free spatiotemporal attention mechanism for video question answering,” IEEE Transactions on Multimedia , 2023
2023
Cited alongside, same era.

Then

A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. l. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier et al. , “Mistral 7b,” arXiv preprint arXiv:2310.06825 , 2023
Original
2023
Closest in time.
L. Momeni, M. Caron, A. Nagrani, A. Zisserman, and C. Schmid, “Verbs in action: Improving verb understanding in video-language models,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 15 579–15 591
2023
Closest in time.
Y. Luo, X. Zheng, X. Yang, G. Li, H. Lin, J. Huang, J. Ji, F. Chao, J. Luo, and R. Ji, “Video-rag: Visually-aligned retrieval-augmented long video comprehension,” arXiv preprint arXiv:2411.13093 , 2024
Original
2024
Closest in time.
Y. Weng, M. Han, H. He, X. Chang, and B. Zhuang, “Longvlm: Efficient long video understanding via large language models,” in European Conference on Computer Vision . Springer, 2024, pp. 453–470
2024
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

Long Video Understanding with Learnable Retrieval in Video-Language Models · Around

Built on

R. F. Thompson and S. A. Madigan, Memory: the key to consciousness . Princeton University Press, 2013, vol. 3
2013
Earlier work this paper cites.
F. Caba Heilbron, V. Escorcia, B. Ghanem, and J. Carlos Niebles, “Activitynet: A large-scale video benchmark for human activity understanding,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2015, pp. 961–970
2015
Earlier work this paper cites.
Z. Yuan, S. Sun, L. Duan, C. Li, X. Wu, and C. Xu, “Adversarial multimodal network for movie story question answering,” IEEE Transactions on Multimedia , vol. 23, pp. 1744–1756, 2020
2020
Earlier work this paper cites.
W. Zhang, S. Tang, Y. Cao, S. Pu, F. Wu, and Y. Zhuang, “Frame augmented alternating attention network for video question answering,” IEEE Transactions on Multimedia , vol. 22, no. 4, pp. 1032–1041, 2020
2020
Earlier work this paper cites.
S. Castro, M. Azab, J. Stroud, C. Noujaim, R. Wang, J. Deng, and R. Mihalcea, “LifeQA: A real-life dataset for video question answering,” in Proceedings of the Twelfth Language Resources and Evaluation Conference , 2020, pp. 4352–4358
2020
Earlier work this paper cites.
J. Wang, B.-K. Bao, and C. Xu, “Dualvgr: A dual-visual graph reasoning unit for video question answering,” IEEE Transactions on Multimedia , vol. 24, pp. 3369–3380, 2021
2021
Earlier work this paper cites.
Z. Guo, J. Zhao, L. Jiao, X. Liu, and F. Liu, “A universal quaternion hypergraph network for multimodal video question answering,” IEEE Transactions on Multimedia , vol. 25, pp. 38–49, 2021
2021
Earlier work this paper cites.
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al. , “Learning transferable visual models from natural language supervision,” in International conference on machine learning . PMLR, 2021, pp. 8748–8763
2021
Earlier work this paper cites.
J. Xiao, X. Shang, A. Yao, and T.-S. Chua, “Next-qa: Next phase of question-answering to explaining temporal actions,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2021, pp. 9777–9786
2021
Earlier work this paper cites.
A. Yang, A. Miech, J. Sivic, I. Laptev, and C. Schmid, “Just ask: Learning to answer questions from millions of narrated videos,” in Proceedings of the IEEE/CVF international conference on computer vision , 2021, pp. 1686–1697
2021
Earlier work this paper cites.
J. Jiang, Z. Liu, and N. Zheng, “Livlr: A lightweight visual-linguistic reasoning framework for video question answering,” IEEE Transactions on Multimedia , vol. 25, pp. 5002–5013, 2022
2022
Earlier work this paper cites.
J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds et al. , “Flamingo: a visual language model for few-shot learning,” Advances in Neural Information Processing Systems , vol. 35, pp. 23 716–23 736, 2022
2022
Earlier work this paper cites.
S. Castro, N. Deng, P. Huang, M. G. Burzo, and R. Mihalcea, “In-the-wild video question answering,” in International Conference on Computational Linguistics , 2022, pp. 5613–5635
2022
Earlier work this paper cites.
L. Bärmann and A. Waibel, “Where did i leave my keys? - episodic-memory-based question answering on egocentric videos,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops , June 2022, pp. 1560–1568
2022
Earlier work this paper cites.
A. Yang, A. Miech, J. Sivic, I. Laptev, and C. Schmid, “Zero-shot video question answering via frozen bidirectional language models,” Advances in Neural Information Processing Systems , vol. 35, pp. 124–141, 2022
2022
Earlier work this paper cites.
Y. Wang, K. Li, Y. Li, Y. He, B. Huang, Z. Zhao, H. Zhang, J. Xu, Y. Liu, Z. Wang et al. , “Internvideo: General video foundation models via generative and discriminative learning,” arXiv preprint arXiv:2212.03191 , 2022
Original
2022
Earlier work this paper cites.
J. Xiao, A. Yao, Z. Liu, Y. Li, W. Ji, and T.-S. Chua, “Video as conditional graph hierarchy for multi-granular question answering,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 36, no. 3, 2022, pp. 2804–2812
2022
Earlier work this paper cites.
J. Xiao, P. Zhou, T.-S. Chua, and S. Yan, “Video graph transformer for video question answering,” 2022. [Online]. Available: https://arxiv.org/abs/2207.05342
Original
2022
Earlier work this paper cites.
L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray et al. , “Training language models to follow instructions with human feedback,” Advances in Neural Information Processing Systems , vol. 35, pp. 27 730–27 744, 2022
2022
Earlier work this paper cites.
M. Maaz, H. Rasheed, S. Khan, and F. S. Khan, “Video-ChatGPT: Towards detailed video understanding via large vision and language models,” arXiv preprint arXiv:2306.05424 , 2023
Original
2023
Earlier work this paper cites.
J. Wang, D. Chen, C. Luo, X. Dai, L. Yuan, Z. Wu, and Y.-G. Jiang, “Chatvideo: A tracklet-centric multimodal and versatile video understanding system,” arXiv preprint arXiv:2304.14407 , 2023
Original
2023
Earlier work this paper cites.

Similar

K. Li, Y. He, Y. Wang, Y. Li, W. Wang, P. Luo, Y. Wang, L. Wang, and Y. Qiao, “Videochat: Chat-centric video understanding,” arXiv preprint arXiv:2305.06355 , 2023
Original
2023
Cited alongside, same era.
H. Zhang, X. Li, and L. Bing, “Video-LLaMA: An instruction-tuned audio-visual language model for video understanding,” arXiv preprint arXiv:2306.02858 , 2023
Original
2023
Cited alongside, same era.
T. Qian, R. Cui, J. Chen, P. Peng, X. Guo, and Y.-G. Jiang, “Locate before answering: Answer guided question localization for video question answering,” IEEE Transactions on Multimedia , 2023
2023
Cited alongside, same era.
F. Zhang, R. Wang, F. Zhou, Y. Luo, and J. Li, “Psam: Parameter-free spatiotemporal attention mechanism for video question answering,” IEEE Transactions on Multimedia , 2023
2023
Cited alongside, same era.

Then

A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. l. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier et al. , “Mistral 7b,” arXiv preprint arXiv:2310.06825 , 2023
Original
2023
Closest in time.
L. Momeni, M. Caron, A. Nagrani, A. Zisserman, and C. Schmid, “Verbs in action: Improving verb understanding in video-language models,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 15 579–15 591
2023
Closest in time.
Y. Luo, X. Zheng, X. Yang, G. Li, H. Lin, J. Huang, J. Ji, F. Chao, J. Luo, and R. Ji, “Video-rag: Visually-aligned retrieval-augmented long video comprehension,” arXiv preprint arXiv:2411.13093 , 2024
Original
2024
Closest in time.
Y. Weng, M. Han, H. He, X. Chang, and B. Zhuang, “Longvlm: Efficient long video understanding via large language models,” in European Conference on Computer Vision . Springer, 2024, pp. 453–470
2024
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…