Fetching the paper…
Reading the bibliography…
Automation of existing Graphical User Interfaces (GUIs) is important but hard to achieve.
Modeling Mobile Interface Tappability Using Crowdsourcing and Deep Learning
Amanda Swearngin and Yang Li. 2019 · 1902
Earlier work this paper cites.
Developing a strategy for" Battleship"
EY Rodin, J Cowley, K Huck, S Payne, and D Politte. 1988 · 1988
Earlier work this paper cites.
Cellular Automata: Theory and Experiment
Howard Gutowitz. 1991 · 1991
Earlier work this paper cites.
Watch What I Do: Programming by Demonstration
Allen Cypher, Daniel C. Halbert, David Kurlander, Henry Lieberman, David Maulsby, Brad A. Myers, and Alan Turransky (Eds.). 1993 · 1993
Earlier work this paper cites.
Your Wish is My Command: Programming by Example
2001 · 2001
Earlier work this paper cites.
Learning a similarity metric discriminatively, with application to face verification. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05) , Vol. 1. 539–546 vol. 1
S. Chopra, R. Hadsell, and Y. LeCun. 2005 · 2005
Earlier work this paper cites.
Dimensionality Reduction by Learning an Invariant Mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06) , Vol. 2. 1735–1742
R. Hadsell, S. Chopra, and Y. LeCun. 2006 · 2006
Earlier work this paper cites.
Koala: capture, share, automate, personalize business processes on the web. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (, San Jose, California, USA,) (CHI ’07) . Association for Computing Machinery, New York, NY, USA, 943–946
Greg Little, Tessa A. Lau, Allen Cypher, James Lin, Eben M. Haber, and Eser Kandogan. 2007 · 2007
Earlier work this paper cites.
CoScripter: automating & sharing how-to knowledge in the enterprise. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Florence, Italy) (CHI ’08) . Association for Computing Machinery, New York, NY, USA, 1719–1728
Gilly Leshed, Eben M. Haber, Tara Matthews, and Tessa Lau. 2008 · 2008
Earlier work this paper cites.
Sikuli: Using GUI Screenshots for Search and Automation. In Proceedings of the 22nd Annual ACM Symposium on User Interface Software and Technology (Victoria, BC, Canada) (UIST ’09) . Association for Computing Machinery, New York, NY, USA, 183–192
Tom Yeh, Tsung-Hsiang Chang, and Robert C. Miller. 2009 · 2009
Earlier work this paper cites.
Prefab: Implementing Advanced Behaviors Using Pixel-Based Reverse Engineering of Interface Structure. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Atlanta, Georgia, USA) (CHI ’10) . Association for Computing Machinery, New York, NY, USA, 1525–1534
Morgan Dixon and James Fogarty. 2010 · 2010
Earlier work this paper cites.
Here’s what I did: sharing and reusing web activity with ActionShot. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Atlanta, Georgia, USA) (CHI ’10) . Association for Computing Machinery, New York, NY, USA, 723–732
Ian Li, Jeffrey Nichols, Tessa Lau, Clemens Drews, and Allen Cypher. 2010 · 2010
Earlier work this paper cites.
Deep Residual Learning for Image Recognition
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. 2015 · 2015
Earlier work this paper cites.
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. In Advances in Neural Information Processing Systems , C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (Eds.), Vol. 28. Curran Associates, Inc
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015 · 2015
Earlier work this paper cites.
You Only Look Once: Unified, Real-Time Object Detection. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . 779–788
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. 2016 · 2016
Earlier work this paper cites.
Rico: A Mobile App Dataset for Building Data-Driven Design Applications. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (Québec City, QC, Canada) (UIST ’17) . Association for Computing Machinery, New York, NY, USA, 845–854
Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. 2017 · 2017
Earlier work this paper cites.
Help, It Looks Confusing: GUI Task Automation Through Demonstration and Follow-up Questions. In Proceedings of the 22nd International Conference on Intelligent User Interfaces (Limassol, Cyprus) (IUI ’17) . Association for Computing Machinery, New York, NY, USA, 233–243
Thanapong Intharah, Daniyar Turmukhambetov, and Gabriel J. Brostow. 2017 · 2017
Cited alongside, same era.
SUGILITE: Creating Multimodal Smartphone Automation by Demonstration. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’17) . Association for Computing Machinery, New York, NY, USA, 6038–6049
Toby Jia-Jun Li, Amos Azaria, and Brad A. Myers. 2017 · 2017
Cited alongside, same era.
Focal Loss for Dense Object Detection. In 2017 IEEE International Conference on Computer Vision (ICCV) . 2999–3007
Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. 2017b · 2017
Cited alongside, same era.
World of Bits: An Open-Domain Platform for Web-Based Agents. In Proceedings of the 34th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 70) , Doina Precup and Yee Whye Teh (Eds.). PMLR, 3135–3144
Benchmarking automated GUI testing for Android against real-world bugs. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (Athens, Greece) (ESEC/FSE 2021) . Association for Computing Machinery, New York, NY, USA, 119–130
Ting Su, Jue Wang, and Zhendong Su. 2021 · 2021
Later among the works it cites.
Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels. In CHI
Xiaoyi Zhang, Lilian de Greef, Amanda Swearngin, Samuel White, Kyle Murray, Lisa Yu, Qi Shan, Jeffrey Nichols, Jason Wu, Chris Fleizach, Aaron Everitt, and Jeffrey P. Bigham. 2021 · 2021
Later among the works it cites.
Understanding Screen Relationships from Screenshots of Smartphone Applications. In Proceedings of the 27th International Conference on Intelligent User Interfaces (, Helsinki, Finland,) (IUI ’22) . Association for Computing Machinery, New York, NY, USA, 447–458
Shirin Feiz, Jason Wu, Xiaoyi Zhang, Amanda Swearngin, Titus Barik, and Jeffrey Nichols. 2022 · 2022
Later among the works it cites.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang. 2017 · 2017
Cited alongside, same era.
Aggregated Residual Transformations for Deep Neural Networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . 5987–5995
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. 2017 · 2017
Cited alongside, same era.
Reinforcement Learning on Web Interfaces using Workflow-Guided Exploration. In International Conference on Learning Representations
Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, and Percy Liang. 2018 · 2018
Cited alongside, same era.
HILC: Domain-Independent PbD System Via Computer Vision and Follow-Up Questions
Thanapong Intharah, Daniyar Turmukhambetov, and Gabriel J. Brostow. 2019 · 2019
Cited alongside, same era.
PUMICE: A Multi-Modal Agent That Learns Concepts and Conditionals from Natural Language and Demonstrations. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology (New Orleans, LA, USA) (UIST ’19) . Association for Computing Machinery, New York, NY, USA, 577–589
Toby Jia-Jun Li, Marissa Radensky, Justin Jia, Kirielle Singarajah, Tom M. Mitchell, and Brad A. Myers. 2019 · 2019
Cited alongside, same era.
FCOS: Fully Convolutional One-Stage Object Detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)
Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. 2019 · 2019
Cited alongside, same era.
Self-Attention Generative Adversarial Networks. In Proceedings of the 36th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 97) , Kamalika Chaudhuri and Ruslan Salakhutdinov (Eds.). PMLR, 7354–7363
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. 2019 · 2019
Cited alongside, same era.
Translating video recordings of mobile app usages into replayable scenarios. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering (Seoul, South Korea) (ICSE ’20) . Association for Computing Machinery, New York, NY, USA, 309–321
Carlos Bernal-Cárdenas, Nathan Cooper, Kevin Moran, Oscar Chaparro, Andrian Marcus, and Denys Poshyvanyk. 2020 · 2020
Cited alongside, same era.
VASTA: A Vision and Language-Assisted Smartphone Task Automation System. In Proceedings of the 25th International Conference on Intelligent User Interfaces (Cagliari, Italy) (IUI ’20) . Association for Computing Machinery, New York, NY, USA, 22–32
Alborz Rezazadeh Sereshkeh, Gary Leung, Krish Perumal, Caleb Phillips, Minfan Zhang, Afsaneh Fazly, and Iqbal Mohomed. 2020 · 2020
Cited alongside, same era.
Computational Approaches for Understanding, Generating, and Adapting User Interfaces. In Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI EA ’22) . Association for Computing Machinery, New York, NY, USA, Article 74, 6 pages
Yue Jiang, Yuwen Lu, Jeffrey Nichols, Wolfgang Stuerzlinger, Chun Yu, Christof Lutteroth, Yang Li, Ranjitha Kumar, and Toby Jia-Jun Li. 2022 · 2022
Later among the works it cites.
Robust Speech Recognition via Large-Scale Weak Supervision
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2022 · 2022
Later among the works it cites.
Low-Code Programming Models
Martin Hirzel. 2023 · 2023
Later among the works it cites.
The Future of Computational Approaches for Understanding and Adapting User Interfaces. In Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI EA ’23) . Association for Computing Machinery, New York, NY, USA, Article 367, 5 pages
Yue Jiang, Yuwen Lu, Christof Lutteroth, Toby Jia-Jun Li, Jeffrey Nichols, and Wolfgang Stuerzlinger. 2023 · 2023
Later among the works it cites.
Kotlin Docs
Kotlin. 2023 · 2023
Later among the works it cites.
Sikuli Slides
Sikuli Lab. 2013 · 2023
Later among the works it cites.
Spotlight: Mobile UI Understanding using Vision-Language Models with a Focus. In The Eleventh International Conference on Learning Representations
Gang Li and Yang Li. 2023 · 2023
Later among the works it cites.
WebUI: A Dataset for Enhancing Visual UI Understanding with Web Semantics. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI ’23) . Association for Computing Machinery, New York, NY, USA, Article 286, 14 pages
Jason Wu, Siyan Wang, Siman Shen, Yi-Hao Peng, Jeffrey Nichols, and Jeffrey P Bigham. 2023b · 2023
Later among the works it cites.
SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents. In ICLR 2024 Workshop on Large Language Model (LLM) Agents
Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Li YanTao, Jianbing Zhang, and Zhiyong Wu. 2024 · 2024
Later among the works it cites.
Multimodal Web Navigation with Instruction-Finetuned Foundation Models
Hiroki Furuta, Kuang-Huei Lee, Ofir Nachum, Yutaka Matsuo, Aleksandra Faust, Shixiang Shane Gu, and Izzeddin Gur. 2024 · 2024
Later among the works it cites.
Rabbit R1 review: an AI assistant that actually assists
Sean Hollister. 2024 · 2024
Later among the works it cites.
A comprehensive survey on test-time adaptation under distribution shifts
Jian Liang, Ran He, and Tieniu Tan. 2024 · 2024
Later among the works it cites.