2024

GPT-4V(ision) is a Generalist Web Agent, if Grounded

Zheng, Boyuan, Gou, Boyu, Kil, Jihyung et al.

Understand

The recent development on large multimodal models (LMMs), especially GPT-4V(ision) and Gemini, has been quickly expanding the capability boundaries of multimodal models beyond traditional tasks like image captioning and visual question answering.

In this work, we explore the potential of LMMs like GPT-4V as a generalist web agent that can follow natural language instructions to complete tasks on any given website.
We propose SEEACT, a generalist web agent that harnesses the power of LMMs for integrated visual understanding and acting on the web.
We evaluate on the recent MIND2WEB benchmark.

Built on

Referitgame: Referring to objects in photographs of natural scenes
Kazemzadeh, S., Ordonez, V., andre Matten, M., and Berg, T. L · 2014
Earlier work this paper cites.
Microsoft coco: Common objects in context
Lin, T.-Y., Maire, M., Belongie, S. J., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L · 2014
Earlier work this paper cites.
Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models
Plummer, B. A., Wang, L., Cervantes, C. M., Caicedo, J. C., Hockenmaier, J., and Lazebnik, S · 2015
Earlier work this paper cites.
Making the v in vqa matter: Elevating the role of image understanding in visual question answering
Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., and Parikh, D · 2016
Earlier work this paper cites.
World of bits: An open-domain platform for web-based agents
Shi, T., Karpathy, A., Fan, L. J., Hernández, J. Z., and Liang, P · 2017
Earlier work this paper cites.
Scene parsing through ade20k dataset
Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., and Torralba, A · 2017
Earlier work this paper cites.
Reinforcement learning on web interfaces using workflow-guided exploration
Original
Liu, E. Z., Guu, K., Pasupat, P., Shi, T., and Liang, P · 2018
Earlier work this paper cites.
Measuring massive multitask language understanding
Original
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D. X., and Steinhardt, J · 2020
Earlier work this paper cites.
Flin: A flexible natural language interface for web navigation
Original
Mazumder, S. and Riva, O · 2020
Earlier work this paper cites.
Grounding ‘grounding’in nlp
Chandu, K. R., Bisk, Y., and Black, A. W · 2021
Earlier work this paper cites.
Learning transferable visual models from natural language supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al · 2021
Earlier work this paper cites.
How much can clip benefit vision-and-language tasks?
Original
Shen, S., Li, L. H., Tan, H., Bansal, M., Rohrbach, A., Chang, K.-W., Yao, Z., and Keutzer, K · 2021
Earlier work this paper cites.
Flamingo: a visual language model for few-shot learning
Original
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., Ring, R., Rutherford, E., Cabi, S., Han, T., Gong, Z., Samangooei, S., Monteiro, M., Menick, J., Borgeaud, S., Brock, A., Nematzadeh, A., Sharifzadeh, S., Binkowski, M., Barreira, R., Vinyals, O., Zisserman, A., and Simonyan, K · 2022
Earlier work this paper cites.
Scaling instruction-finetuned language models
Original
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, E., Wang, X., Dehghani, M., Brahma, S., Webson, A., Gu, S. S., Dai, Z., Suzgun, M., Chen, X., Chowdhery, A., Valter, D., Narang, S., Mishra, G., Yu, A. W., Zhao, V., Huang, Y., Dai, A. M., Yu, H., Petrov, S., hsin Chi, E. H., Dean, J., Devlin, J., Roberts, A., Zhou, D., Le, Q. V., and Wei, J · 2022
Earlier work this paper cites.
Understanding html with large language models
Gur, I., Nachum, O., Miao, Y., Safdari, M., Huang, A., Chowdhery, A., Narang, S., Fiedel, N., and Faust, A · 2022
Earlier work this paper cites.

Similar

Pix2struct: Screenshot parsing as pretraining for visual language understanding
Original
Lee, K., Joshi, M., Turc, I., Hu, H., Liu, F., Eisenschlos, J. M., Khandelwal, U., Shaw, P., Chang, M.-W., and Toutanova, K · 2022
Cited alongside, same era.
Learn to explain: Multimodal reasoning via thought chains for science question answering
Original
Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., and Kalyan, A · 2022
Cited alongside, same era.
Scienceqa: a novel resource for question answering on scholarly articles
Saikh, T., Ghosal, T., Mittal, A., Ekbal, A., and Bhattacharyya, P · 2022
Cited alongside, same era.
Webshop: Towards scalable real-world web interaction with grounded language agents
Original
Yao, S., Chen, H., Yang, J., and Narasimhan, K · 2022
Cited alongside, same era.
An in-depth look at gemini’s language abilities
Akter, S. N., Yu, Z., Muhamed, A., Ou, T., Bauerle, A., Cabrera, Á. A., Dholakia, K., Xiong, C., and Neubig, G · 2023
Cited alongside, same era.
Gemini: A family of highly capable multimodal models
Anil, G. T. G. R., Borgeaud, S., and et al., Y. W · 2023
Cited alongside, same era.
Shikra: Unleashing multimodal llm’s referential dialogue magic
Original
Chen, K., Zhang, Z., Zeng, W., Zhang, R., Zhu, F., and Zhao, R · 2023
Cited alongside, same era.
Mind2web: Towards a generalist agent for the web
Original
Deng, X., Gu, Y., Zheng, B., Chen, S., Stevens, S., Wang, B., Sun, H., and Su, Y · 2023
Cited alongside, same era.
Multimodal web navigation with instruction-finetuned foundation models
Original
Furuta, H., Nachum, O., Lee, K.-H., Matsuo, Y., Gu, S. S., and Gur, I · 2023
Cited alongside, same era.
Don’t generate, discriminate: A proposal for grounding language models to real-world environments
Gu, Y., Deng, X., and Su, Y · 2023
Cited alongside, same era.
Hallusionbench: An advanced diagnostic suite for entangled language hallucination&visual illusion in large vision-language models
Guan, T., Liu, F., Wu, X., Xian, R., Li, Z., Liu, X., Wang, X., Chen, L., Huang, F., Yacoob, Y., Manocha, D., and Zhou, T · 2023
Cited alongside, same era.
A real-world webagent with planning, long context understanding, and program synthesis
Original
Gur, I., Furuta, H., Huang, A., Safdari, M., Matsuo, Y., Eck, D., and Faust, A · 2023
Cited alongside, same era.

Then

Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models
Original
Li, J., Li, D., Savarese, S., and Hoi, S. C. H · 2023
Later among the works it cites.
Gpt-4 technical report
Original
OpenAI · 2023
Later among the works it cites.
Kosmos-2: Grounding multimodal large language models to the world
Original
Peng, Z., Wang, W., Dong, L., Hao, Y., Huang, S., Ma, S., and Wei, F · 2023
Later among the works it cites.
From pixels to ui actions: Learning to follow instructions via graphical user interfaces
Original
Shaw, P., Joshi, M., Cohan, J., Berant, J., Pasupat, P., Hu, H., Khandelwal, U., Lee, K., and Toutanova, K · 2023
Later among the works it cites.
What does clip know about a red circle? visual prompt engineering for vlms
Original
Shtedritski, A., Rupprecht, C., and Vedaldi, A · 2023
Later among the works it cites.
Hierarchical prompting assists large language model on web navigation
Original
Sridhar, A., Lo, R., Xu, F. F., Zhu, H., and Zhou, S · 2023
Later among the works it cites.
Gpt-4v in wonderland: Large multimodal models for zero-shot smartphone gui navigation
Yan, A., Yang, Z., Zhu, W., Lin, K., Li, L., Wang, J., Yang, J., Zhong, Y., McAuley, J., Gao, J., Liu, Z., and Wang, L · 2023
Later among the works it cites.
Ferret: Refer and ground anything anywhere at any granularity
Original
You, H., Zhang, H., Gan, Z., Du, X., Zhang, B., Wang, Z., Cao, L., Chang, S.-F., and Yang, Y · 2023
Later among the works it cites.
Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi
Original
Yue, X., Ni, Y., Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., Wei, C., Yu, B., Yuan, R., Sun, R., Yin, M., Zheng, B., Yang, Z., Liu, Y., Huang, W., Sun, H., Su, Y., and Chen, W · 2023
Later among the works it cites.
Gpt-4v(ision) as a generalist evaluator for vision-language tasks
Original
Zhang, X., Lu, Y., Wang, W., Yan, A., Yan, J., Qin, L., Wang, H., Yan, X., Wang, W. Y., and Petzold, L. R · 2023
Later among the works it cites.
Bubogpt: Enabling visual grounding in multi-modal llms
Original
Zhao, Y., Lin, Z., Zhou, D., Huang, Z., Feng, J., and Kang, B · 2023
Later among the works it cites.
Agieval: A human-centric benchmark for evaluating foundation models
Original
Zhong, W., Cui, R., Guo, Y., Liang, Y., Lu, S., Wang, Y., Saied, A. S. S., Chen, W., and Duan, N · 2023
Later among the works it cites.
Seeclick: Harnessing gui grounding for advanced visual gui agents
Cheng, K., Sun, Q., Chu, Y., Xu, F., Li, Y., Zhang, J., and Wu, Z · 2024
Closest in time.
Visualwebarena: Evaluating multimodal agents on realistic visual web tasks
Koh, J. Y., Lo, R., Jang, L., Duvvur, V., Lim, M. C., Huang, P.-Y., Neubig, G., Zhou, S., Salakhutdinov, R., and Fried, D · 2024
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…