Fetching the paper…

Reading the bibliography…

2025

WebNav: An Intelligent Agent for Voice-Controlled Web Navigation

Srinivasan, Trisanth, Patapati, Santosh

Original
PDF
Post on X

Understand

The current state of modern web interfaces, especially in regards to accessibility focused usage is extremely lacking.

Traditional methods for web interaction, such as scripting languages and screen readers, often lack the flexibility to handle dynamic content or the intelligence to interpret high-level user goals.
To address these limitations, we introduce WebNav, a novel agent for multi-modal web navigation.
WebNav leverages a dual Large Language Model (LLM) architecture to translate natural language commands into precise, executable actions on a graphical user interface.

WebNav: An Intelligent Agent for Voice-Controlled Web Navigation · Around

Built on

J. Hailpern, R. Ladner, A. Lisin et al. , “Web 2.0: Blind to an accessible new world,” in Proc. 18th Int’l ACM SIGACCESS Conf. on Computers and Accessibility (W4A) , 2009, pp. 1–8
2009
Earlier work this paper cites.
W3C Web Accessibility Initiative, “Web content accessibility guidelines (wcag) 2.1,” 2018, w3C Recommendation; https://www.w3.org/TR/WCAG21/
2018
Earlier work this paper cites.
W. H. Organization, World Report on Vision . World Health Organization, 2019
2019
Earlier work this paper cites.
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, N. Minderer, G. Heigold, S. Gelly, J. Houlsby, and O. Vinyals, “An image is worth 16x16 words: Transformers for image recognition at scale,” in Proc. Int. Conf. on Learning Representations (ICLR) , 2021
2021
Earlier work this paper cites.
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al. , “Learning transferable visual models from natural language supervision,” in Proc. Int. Conf. on Machine Learning (ICML) , 2021
2021
Earlier work this paper cites.
A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H.-W. Chung, C. Sutton et al. , “Palm: Scaling language modeling with pathways,” arXiv preprint arXiv:2204.02311 , 2022
Original
2022
Earlier work this paper cites.
J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Chi, S. Le, H. Zhou, D. Song, J. Feng, D. Zhang et al. , “Chain-of-thought prompting elicits reasoning in large language models,” in Advances in Neural Information Processing Systems 35 , 2022, pp. 24 824–24 837
2022
Earlier work this paper cites.
A. Radford, J. Wu, R. Child, D. Luan, T. Amodei, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” arXiv preprint arXiv:2212.04356 , 2022
Original
2022
Earlier work this paper cites.

Similar

S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y.-T. Lee, Y. Li, S. Lundberg et al. , “Sparks of artificial general intelligence: Early experiments with gpt-4,” arXiv preprint arXiv:2303.12712 , 2023
Original
2023
Cited alongside, same era.
S. Yao, R. Ye, R. Chintala, C. Dathathri, J. Sharma, A. Jain, and P. Liang, “Tree of thoughts: Deliberate problem solving with large language models,” in Advances in Neural Information Processing Systems (NeurIPS) 36 , 2023, pp. 4110–4126
2023
Cited alongside, same era.
Z. Yu, J. Wei, K. Wang, and et al., “Application of large language models in mobile app test generation,” arXiv preprint arXiv:2307.04724 , 2023
Original
2023
Cited alongside, same era.
X. Deng, Y.-C. Hsiao, H. Behl, S. Ghosh, S. Iyer, P.-C. Hsieh, R. Prasaath, S. Gummadi, C. Li, M. Sundaram, J. Gao, K. Narasimhan, R. Agarwal, H. Firooz, G. Bansal, T.-K. S. Kumar, and D. Lange, “Mind2web: Towards a generalist agent for the web,” in Advances in Neural Information Processing Systems (NeurIPS) 36 , 2023, pp. 22 179–22 191

Then

“Playwright documentation,” https://playwright.dev/docs/intro , 2024, [Online; accessed 21-May-2024]
2024
Later among the works it cites.
Freedom Scientific, “Jaws (job access with speech),” https://www.freedomscientific.com/Products/Blindness/JAWS , 2024, [Online; accessed 21-May-2024]
2024
Later among the works it cites.
NV Access, “Nonvisual desktop access (nvda),” https://www.nvaccess.org/ , 2024, [Online; accessed 21-May-2024]
2024
Later among the works it cites.
M. Ferreira, L. Viegas, J. P. Faria, and B. Lima, “Acceptance test generation with large language models: An industrial case study,” arXiv preprint arXiv:2504.07244 , 2025, presented at SPSNA 2024 workshop; supplementary materials available
Original
2024
Later among the works it cites.
H. Furuta, K.-H. Lee, O. Nachum, J. Hwang, S. Jin, L. Song, and D.-Y. Kim, “Webgum: Multimodal web navigation with instruction-finetuned foundation models,” arXiv preprint arXiv:2305.11854 , 2023, accepted at ICLR 2024

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

2023

Cited alongside, same era.

OpenAI, “Gpt-4v(ision) system card,” Tech. Rep., 2023, arXiv preprint arXiv:2309.17421

2023

Cited alongside, same era.

S. Yao, Y. Zhao, D. Bommasani, and P. Liang, “React: Synergizing reasoning and acting in language models,” in Advances in Neural Information Processing Systems (NeurIPS) 36 , 2023, pp. 26 639–26 653

2023

Cited alongside, same era.

B. Tang and H. Shin, “Steward: Natural language web automation,” arXiv preprint arXiv:2409.15441 , 2024

2024

Cited alongside, same era.

“Selenium webdriver documentation,” https://www.selenium.dev/documentation/webdriver/ , 2024, [Online; accessed 21-May-2024]

2024

Cited alongside, same era.

2024

Later among the works it cites.

H. Lai, X. Liu, I. Iong, S. Yao, Y. Chen, P. Shen, H. Yu, H. Zhang, X. Zhang, Y. Dong, and J. Tang, “Autowebglm: A large language model-based web navigating agent,” in Proc. 30th ACM SIGKDD Int’l Conf. on Knowledge Discovery and Data Mining (KDD) , 2024, pp. —

2024

Later among the works it cites.

A. Sherifi, C. Pan, S. Ding, S. Pan, and D. Lo, “A survey on the application of large language models in software testing,” arXiv preprint arXiv:2501.00217 , 2025

2025

Closest in time.

T. Huang, M. Basu, T. Ward, and D. Khashabi, “R2d2: Remembering, reflecting, and dynamic decision making for web agents,” arXiv preprint arXiv:2501.12485 , 2025

2025

Closest in time.