MobileVLM: A Vision-Language Model for Better Intra- and Inter-UI Understanding

The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale

Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. 2020 · 1981

Earlier work this paper cites.

Mapping natural language instructions to mobile ui action sequences

Original

Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. 2020 · 2005

Earlier work this paper cites.

The probabilistic relevance framework: Bm25 and beyond

Stephen Robertson, Hugo Zaragoza, et al. 2009 · 2009

Earlier work this paper cites.

An image is worth 16x16 words: Transformers for image recognition at scale

Original

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020 · 2010

Earlier work this paper cites.

Coco-text: Dataset and benchmark for text detection and recognition in natural images

Original

Andreas Veit, Tomas Matera, Lukas Neumann, Jiri Matas, and Serge Belongie. 2016 · 2016

Earlier work this paper cites.

Rico: A mobile app dataset for building data-driven design applications

Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. 2017 · 2017

Earlier work this paper cites.

Objects365: A large-scale, high-quality dataset for object detection

Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. 2019 · 2019

Earlier work this paper cites.

Uibert: Learning generic multimodal representations for ui understanding

Original

Chongyang Bai, Xiaoxue Zang, Ying Xu, Srinivas Sunkara, Abhinav Rastogi, Jindong Chen, et al. 2021 · 2021

Earlier work this paper cites.

Mobile app tasks with iterative feedback (motif): Addressing task feasibility in interactive visual environments

Original

Andrea Burns, Deniz Arsan, Sanjna Agrawal, Ranjitha Kumar, Kate Saenko, and Bryan A Plummer. 2021 · 2021

Earlier work this paper cites.

Boundary iou: Improving object-centric image segmentation evaluation

Bowen Cheng, Ross Girshick, Piotr Dollár, Alexander C Berg, and Alexander Kirillov. 2021 · 2021

Earlier work this paper cites.

Vut: Versatile ui transformer for multi-modal multi-task user interface modeling

Original

Yang Li, Gang Li, Xin Zhou, Mostafa Dehghani, and Alexey Gritsenko. 2021 · 2021

Earlier work this paper cites.

Screen recognition: Creating accessibility metadata for mobile applications from pixels

Xiaoyi Zhang, Lilian De Greef, Amanda Swearngin, Samuel White, Kyle Murray, Lisa Yu, Qi Shan, Jeffrey Nichols, Jason Wu, Chris Fleizach, et al. 2021b · 2021

Cited alongside, same era.

Coyo-700m: Image-text pair dataset

Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Saehoon Kim. 2022 · 2022

Cited alongside, same era.

Screenqa: Large-scale question-answer pairs over mobile app screenshots

Original

Yu-Chung Hsiao, Fedir Zubach, Maria Wang, et al. 2022 · 2022

Cited alongside, same era.

Laion-5b: An open large-scale dataset for training next generation image-text models

Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. 2022 · 2022

Cited alongside, same era.

Meta-gui: towards multi-modal conversational agents on mobile gui

Original

Liangtai Sun, Xingyu Chen, Lu Chen, Tianle Dai, Zichen Zhu, and Kai Yu. 2022 · 2022

Empowering llm to use smartphone for intelligent task automation

Original

Hao Wen, Yuanchun Li, Guohong Liu, Shanhui Zhao, Tao Yu, Toby Jia-Jun Li, Shiqi Jiang, Yunhao Liu, Yaqin Zhang, and Yunxin Liu. 2023 · 2023

Later among the works it cites.

Gpt-4v in wonderland: Large multimodal models for zero-shot smartphone gui navigation

Original

An Yan, Zhengyuan Yang, Wanrong Zhu, Kevin Lin, Linjie Li, Jianfeng Wang, Jianwei Yang, Yiwu Zhong, Julian McAuley, Jianfeng Gao, et al. 2023 · 2023

Later among the works it cites.

Appagent: Multimodal agents as smartphone users

Original

Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. 2023 · 2023

Later among the works it cites.

You only look at screens: Multimodal chain-of-action agents

Original

Zhuosheng Zhan and Aston Zhang. 2023 · 2023

Later among the works it cites.