Fetching the paper…
Reading the bibliography…
Modeling user interfaces (UIs) from visual information allows systems to make inferences about the functionality and semantics needed to support use cases in accessibility, app automation, and testing.
Dropout: a simple way to prevent neural networks from overfitting
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014 · 1958
Earlier work this paper cites.
Making the GUI Talk
Richard S. Schwerdtfeger. 1991 · 1991
Earlier work this paper cites.
SMOTE: synthetic minority over-sampling technique
Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. 2002 · 2002
Earlier work this paper cites.
Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06) , Vol. 2. IEEE, 1735–1742
Raia Hadsell, Sumit Chopra, and Yann LeCun. 2006 · 2006
Earlier work this paper cites.
Semi-supervised learning
Olivier Chapelle, Bernhard Scholkopf, and Alexander Zien. 2009 · 2009
Earlier work this paper cites.
Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition . Ieee, 248–255
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009 · 2009
Earlier work this paper cites.
A survey on transfer learning
Sinno Jialin Pan and Qiang Yang. 2009 · 2009
Earlier work this paper cites.
Sikuli: using GUI screenshots for search and automation. In Proceedings of the 22nd annual ACM symposium on User interface software and technology . 183–192
Tom Yeh, Tsung-Hsiang Chang, and Robert C Miller. 2009 · 2009
Earlier work this paper cites.
Prefab: implementing advanced behaviors using pixel-based reverse engineering of interface structure. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems . 1525–1534
Morgan Dixon and James Fogarty. 2010 · 2010
Earlier work this paper cites.
Associating the visual representation of user interfaces with their internal structures and metadata. In Proceedings of the 24th annual ACM symposium on User interface software and technology . 245–256
Tsung-Hsiang Chang, Tom Yeh, and Rob Miller. 2011 · 2011
Earlier work this paper cites.
Webzeitgeist: design mining the web. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems . ACM, New York, NY, USA, 3083–3092
Ranjitha Kumar, Arvind Satyanarayan, Cesar Torres, Maxine Lim, Salman Ahmad, Scott R Klemmer, and Jerry O Talton. 2013 · 2013
Earlier work this paper cites.
Microsoft coco: Common objects in context. In European conference on computer vision . Springer, 740–755
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014 · 2014
Earlier work this paper cites.
Very deep convolutional networks for large-scale image recognition
Karen Simonyan and Andrew Zisserman. 2014 · 2014
Earlier work this paper cites.
ERICA: Interaction mining mobile apps. In Proceedings of the 29th annual symposium on user interface software and technology . 767–776
Biplab Deka, Zifeng Huang, and Ranjitha Kumar. 2016 · 2016
Earlier work this paper cites.
Understanding how image quality affects deep neural networks. In 2016 eighth international conference on quality of multimedia experience (QoMEX) . IEEE, 1–6
Samuel Dodge and Lina Karam. 2016 · 2016
Earlier work this paper cites.
Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition . 770–778
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016 · 2016
Earlier work this paper cites.
Deep networks with stochastic depth. In European conference on computer vision . Springer, 646–661
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. 2016 · 2016
Earlier work this paper cites.
Ssd: Single shot multibox detector. In European conference on computer vision . Springer, 21–37
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C Berg. 2016 · 2016
Earlier work this paper cites.
Rico: A mobile app dataset for building data-driven design applications. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology . 845–854
Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. 2017 · 2017
Cited alongside, same era.
SUGILITE: creating multimodal smartphone automation by demonstration. In Proceedings of the 2017 CHI conference on human factors in computing systems . 6038–6049
Toby Jia-Jun Li, Amos Azaria, and Brad A Myers. 2017a · 2017
Cited alongside, same era.
Droidbot: a lightweight ui-guided test input generator for android. In 2017 IEEE/ACM 39th International Conference on Software Engineering Companion (ICSE-C) . IEEE, 23–26
Yuanchun Li, Ziyue Yang, Yao Guo, and Xiangqun Chen. 2017b · 2017
Cited alongside, same era.
Learning design semantics for mobile apps. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology . 569–579
Thomas F Liu, Mark Craft, Jason Situ, Ersin Yumer, Radomir Mech, and Ranjitha Kumar. 2018 · 2018
Cited alongside, same era.
Actionbert: Leveraging user actions for semantic understanding of user interfaces. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 35. 5931–5938
Zecheng He, Srinivas Sunkara, Xiaoxue Zang, Ying Xu, Lijuan Liu, Nevan Wichers, Gabriel Schubiner, Ruby Lee, and Jindong Chen. 2021 · 2021
Later among the works it cites.
Multibench: Multiscale benchmarks for multimodal representation learning
Paul Pu Liang, Yiwei Lyu, Xiang Fan, Zetian Wu, Yun Cheng, Jason Wu, Leslie Chen, Peter Wu, Michelle A Lee, Yuke Zhu, et al · 2021
Later among the works it cites.
Synz: Enhanced synthetic dataset for training ui element detectors. In 26th International Conference on Intelligent User Interfaces-Companion . 67–69
Vinoth Pandian Sermuga Pandian, Sarah Suleri, and Matthias Jarke. 2021 · 2021
Later among the works it cites.
Screen2words: Automatic mobile UI summarization with multimodal learning. In The 34th Annual ACM Symposium on User Interface Software and Technology . 498–510
Bryan Wang, Gang Li, Xin Zhou, Zhourong Chen, Tovi Grossman, and Yang Li. 2021 · 2021
Later among the works it cites.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Never-ending learning
Tom Mitchell, William Cohen, Estevam Hruschka, Partha Talukdar, Bishan Yang, Justin Betteridge, Andrew Carlson, Bhavana Dalvi, Matt Gardner, Bryan Kisiel, et al · 2018
Cited alongside, same era.
Machine learning-based prototyping of graphical user interfaces for mobile apps
Kevin Moran, Carlos Bernal-Cárdenas, Michael Curcio, Richard Bonett, and Denys Poshyvanyk. 2018a · 2018
Cited alongside, same era.
Humanoid: A deep learning-based approach to automated black-box android app testing. In 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE, 1070–1073
Yuanchun Li, Ziyue Yang, Yao Guo, and Xiangqun Chen. 2019 · 2019
Cited alongside, same era.
Modeling mobile interface tappability using crowdsourcing and deep learning. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems . 1–11
Amanda Swearngin and Yang Li. 2019 · 2019
Cited alongside, same era.
Fcos: Fully convolutional one-stage object detection. In Proceedings of the IEEE/CVF international conference on computer vision . 9627–9636
Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. 2019 · 2019
Cited alongside, same era.
Billion-scale semi-supervised learning for image classification
I Zeki Yalniz, Hervé Jégou, Kan Chen, Manohar Paluri, and Dhruv Mahajan. 2019 · 2019
Cited alongside, same era.
Object detection for graphical user interface: Old fashioned or deep learning or a combination?. In proceedings of the 28th ACM joint meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering . 1202–1214
Jieshan Chen, Mulong Xie, Zhenchang Xing, Chunyang Chen, Xiwei Xu, Liming Zhu, and Guoqiang Li. 2020 · 2020
Cited alongside, same era.
The pile: An 800gb dataset of diverse text for language modeling
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al · 2020
Cited alongside, same era.
Screen Parsing: Towards Reverse Engineering of UI Models from Screenshots. In The 34th Annual ACM Symposium on User Interface Software and Technology . 470–483
Jason Wu, Xiaoyi Zhang, Jeff Nichols, and Jeffrey P Bigham. 2021 · 2021
Later among the works it cites.
Screen recognition: Creating accessibility metadata for mobile applications from pixels. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems . 1–15
Xiaoyi Zhang, Lilian de Greef, Amanda Swearngin, Samuel White, Kyle Murray, Lisa Yu, Qi Shan, Jeffrey Nichols, Jason Wu, Chris Fleizach, et al · 2021
Later among the works it cites.
AutoIt Function PixelSearch
2022 · 2022
Later among the works it cites.
Chrome DevTools engineering blog Full Accessibility Tree in Chrome DevTools
2022 · 2022
Later among the works it cites.
Puppeteer - Chrome
2022 · 2022
Later among the works it cites.
What is the ideal screen size for responsive design?
2022 · 2022
Later among the works it cites.
Translating Video Recordings of Complex Mobile App UI Gestures Into Replayable Scenarios
Carlos Bernal-Cárdenas, Nathan Cooper, Madeleine Havranek, Kevin Moran, Oscar Chaparro, Denys Poshyvanyk, and Andrian Marcus. 2022 · 2022
Later among the works it cites.
Interactive Mobile App Navigation with Uncertain or Under-specified Natural Language Commands
Andrea Burns, Deniz Arsan, Sanjna Agrawal, Ranjitha Kumar, Kate Saenko, and Bryan A Plummer. 2022 · 2022
Later among the works it cites.
Towards Complete Icon Labeling in Mobile Applications. In CHI Conference on Human Factors in Computing Systems . 1–14
Jieshan Chen, Amanda Swearngin, Jason Wu, Titus Barik, Jeffrey Nichols, and Xiaoyi Zhang. 2022 · 2022
Later among the works it cites.
Understanding Screen Relationships from Screenshots of Smartphone Applications. In 27th International Conference on Intelligent User Interfaces . 447–458
Shirin Feiz, Jason Wu, Xiaoyi Zhang, Amanda Swearngin, Titus Barik, and Jeffrey Nichols. 2022 · 2022
Later among the works it cites.
A Large-Scale Longitudinal Analysis of Missing Label Accessibility Failures in Android Apps. In CHI Conference on Human Factors in Computing Systems . 1–16
Raymond Fok, Mingyuan Zhong, Anne Spencer Ross, James Fogarty, and Jacob O Wobbrock. 2022 · 2022
Later among the works it cites.
Learning to Denoise Raw Mobile UI Layouts for Improving Datasets at Scale. In CHI Conference on Human Factors in Computing Systems . 1–13
Gang Li, Gilles Baechler, Manuel Tragut, and Yang Li. 2022 · 2022
Later among the works it cites.
Domain-adversarial training of neural networks
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. 2016 · 2030
Closest in time.