Fetching the paper…
Reading the bibliography…
We use a deep learning based approach to predict whether a selected element in a mobile UI screenshot will be perceived by users as tappable, based on pixels only instead of view hierarchies required by previous work.
Learning from strangers: The art and method of qualitative interview studies
Robert Stuart Weiss. 1994 · 1994
Earlier work this paper cites.
Learning in the Presence of Concept Drift and Hidden Contexts
Gerhard Widmer and Miroslav Kubat. 1996 · 1996
Earlier work this paper cites.
Deploying CogTool: integrating quantitative usability assessment into real-world software development. In 2011 33rd International Conference on Software Engineering (ICSE) . 691–700
Rachel Bellamy, Bonnie John, and Sandra Kogan. 2011 · 2011
Earlier work this paper cites.
ActionBert: Leveraging User Actions for Semantic Understanding of User Interfaces
Zecheng He, Srinivas Sunkara, Xiaoxue Zang, Ying Xu, Lijuan Liu, Nevan Wichers, Gabriel Schubiner, Ruby B. Lee, and Jindong Chen. 2020 · 2012
Earlier work this paper cites.
Tutorons: Generating context-relevant, on-demand explanations and demonstrations of online code. In 2015 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) . 3–12
Andrew Head, Codanda Appachu, Marti A. Hearst, and Björn Hartmann. 2015 · 2015
Earlier work this paper cites.
Beyond Blue Links: Making Clickable Elements Recognizable
Hoa Loranger. 2015 · 2015
Earlier work this paper cites.
Learning through the variation theory: A case study
W. L. Cheng. 2016 · 2016
Earlier work this paper cites.
ERICA: Interaction Mining Mobile Apps. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (Tokyo, Japan) (UIST ’16) . Association for Computing Machinery, New York, NY, USA, 767–776
Biplab Deka, Zifeng Huang, and Ranjitha Kumar. 2016 · 2016
Earlier work this paper cites.
Deep Residual Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . 770–778
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016 · 2016
Earlier work this paper cites.
"Why Should I Trust You?": Explaining the Predictions of Any Classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (San Francisco, California, USA) (KDD ’16) . Association for Computing Machinery, New York, NY, USA, 1135–1144
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016 · 2016
Earlier work this paper cites.
Learning Visual Importance for Graphic Designs and Data Visualizations. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology . ACM, Québec City QC Canada, 57–69
Zoya Bylinskii, Nam Wook Kim, Peter O’Donovan, Sami Alsheikh, Spandan Madan, Hanspeter Pfister, Fredo Durand, Bryan Russell, and Aaron Hertzmann. 2017 · 2017
Earlier work this paper cites.
UI X-Ray: Interactive Mobile UI Testing Based on Computer Vision. In Proceedings of the 22nd International Conference on Intelligent User Interfaces . ACM, Limassol Cyprus, 245–255
Chun-Fu (Richard) Chen, Marco Pistoia, Conglei Shi, Paolo Girolami, Joseph W. Ligman, and Yong Wang. 2017 · 2017
Earlier work this paper cites.
The Mythos of Model Interpretability
Zachary C. Lipton. 2017 · 2017
Earlier work this paper cites.
A Unified Approach to Interpreting Model Predictions. In Proceedings of the 31st International Conference on Neural Information Processing Systems (Long Beach, California, USA) (NIPS’17) . Curran Associates Inc., Red Hook, NY, USA, 4768–4777
Scott M. Lundberg and Su-In Lee. 2017 · 2017
Earlier work this paper cites.
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization. In 2017 IEEE International Conference on Computer Vision (ICCV) . 618–626
Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. 2017 · 2017
Earlier work this paper cites.
Axiomatic Attribution for Deep Networks. In Proceedings of the 34th International Conference on Machine Learning - Volume 70 (Sydney, NSW, Australia) (ICML’17) . JMLR.org, 3319–3328
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017 · 2017
Cited alongside, same era.
Sanity Checks for Saliency Maps. In Advances in Neural Information Processing Systems , S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Eds.), Vol. 31. Curran Associates, Inc
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. 2018 · 2018
Cited alongside, same era.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. 2018 · 2018
Cited alongside, same era.
Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV). In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018 (Proceedings of Machine Learning Research, Vol. 80) , Jennifer G. Dy and Andreas Krause (Eds.). PMLR, 2673–2682
GUIComp: A GUI Design Assistant with Real-Time, Multi-Faceted Feedback
Chunggi Lee, Sanghoon Kim, Dongyun Han, Hongjun Yang, Young-Woo Park, Bum Chul Kwon, and Sungahn Ko. 2020 · 2020
Later among the works it cites.
Enrico: A Dataset for Topic Modeling of Mobile UI Designs. In 22nd International Conference on Human-Computer Interaction with Mobile Devices and Services (Oldenburg, Germany) (MobileHCI ’20) . Association for Computing Machinery, New York, NY, USA, Article 9, 4 pages
Luis A. Leiva, Asutosh Hota, and Antti Oulasvirta. 2020 · 2020
Later among the works it cites.
Mapping Natural Language Instructions to Mobile UI Action Sequences. In Annual Conference of the Association for Computational Linguistics (ACL 2020)
Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. 2020a · 2020
Later among the works it cites.
Artificial Intelligence for HCI: A Modern Approach. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI EA ’20) . Association for Computing Machinery, New York, NY, USA, 1–8
Yang Li, Ranjitha Kumar, Walter S. Lasecki, and Otmar Hilliges. 2020b · 2020
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie J. Cai, James Wexler, Fernanda B. Viégas, and Rory Sayres. 2018 · 2018
Cited alongside, same era.
Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning
Nicolas Papernot and Patrick D. McDaniel. 2018 · 2018
Cited alongside, same era.
Investigating How Experienced UX Designers Effectively Work with Machine Learning. In Proceedings of the 2018 Designing Interactive Systems Conference (Hong Kong, China) (DIS ’18) . Association for Computing Machinery, New York, NY, USA, 585–596
Qian Yang, Alex Scuito, John Zimmerman, Jodi Forlizzi, and Aaron Steinfeld. 2018 · 2018
Cited alongside, same era.
The Effects of Example-Based Explanations in a Machine Learning Interface. In Proceedings of the 24th International Conference on Intelligent User Interfaces (Marina del Ray, California) (IUI ’19) . Association for Computing Machinery, New York, NY, USA, 258–262
Carrie J. Cai, Jonas Jongejan, and Jess Holbrook. 2019 · 2019
Cited alongside, same era.
Gallery D.C.: Design Search and Knowledge Discovery through Auto-created GUI Component Gallery
Chunyang Chen, Sidong Feng, Zhenchang Xing, Linda Liu, Shengdong Zhao, and Jinshui Wang. 2019 · 2019
Cited alongside, same era.
Towards Automatic Concept-based Explanations. In Advances in Neural Information Processing Systems , H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.), Vol. 32. Curran Associates, Inc
Amirata Ghorbani, James Wexler, James Y Zou, and Been Kim. 2019 · 2019
Cited alongside, same era.
Swire: Sketch-Based User Interface Retrieval
Forrest Huang, John F. Canny, and Jeffrey Nichols. 2019 · 2019
Cited alongside, same era.
XRAI: Better Attributions Through Regions. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV) . IEEE, 4947–4956
Andrei Kapishnikov, Tolga Bolukbasi, Fernanda Viegas, and Michael Terry. 2019 · 2019
Cited alongside, same era.
Modeling Mobile Interface Tappability Using Crowdsourcing and Deep Learning. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems . ACM, Glasgow Scotland Uk, 1–11
Amanda Swearngin and Yang Li. 2019 · 2019
Cited alongside, same era.
Later among the works it cites.
Scout: Rapid Exploration of Interface Layout Alternatives through High-Level Design Constraints
Amanda Swearngin, Chenglong Wang, Alannah Oleson, James Fogarty, and Amy J. Ko. 2020 · 2020
Later among the works it cites.
Predicting and Diagnosing User Engagement with Mobile UI Animation via a Data-Driven Approach. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems . ACM, Honolulu HI USA, 1–13
Ziming Wu, Yulun Jiang, Yiding Liu, and Xiaojuan Ma. 2020 · 2020
Later among the works it cites.
UIBert: Learning Generic Multimodal Representations for UI Understanding
Chongyang Bai, Xiaoxue Zang, Ying Xu, Srinivas Sunkara, Abhinav Rastogi, Jindong Chen, and Blaise Agüera y Arcas. 2021 · 2021
Later among the works it cites.
Onboarding Materials as Cross-functional Boundary Objects for Developing AI Assistants. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems . ACM, Yokohama Japan, 1–7
Carrie J Cai, Samantha Winter, David Steiner, Lauren Wilcox, and Michael Terry. 2021 · 2021
Later among the works it cites.
Screen2Vec: Semantic Embedding of GUI Screens and GUI Components
Toby Jia-Jun Li, Lindsay Popowski, Tom Mitchell, and Brad A Myers. 2021 · 2021
Later among the works it cites.
UMLAUT: Debugging Deep Learning Programs Using Program Structure and Model Behavior
Eldon Schoop, Forrest Huang, and Bjoern Hartmann. 2021 · 2021
Later among the works it cites.
Best of both worlds: local and global explanations with human-understandable concepts
Jessica Schrouff, Sebastien Baur, Shaobo Hou, Diana Mincu, Eric Loreaux, Ralph Blanes, James Wexler, Alan Karthikesalingam, and Been Kim. 2021 · 2021
Later among the works it cites.
Screen Parsing: Towards Reverse Engineering of UI Models from Screenshots
Jason Wu, Xiaoyi Zhang, Jeffrey Nichols, and Jeffrey P. Bigham. 2021 · 2021
Later among the works it cites.
Multimodal Icon Annotation For Mobile Applications
Xiaoxue Zang, Ying Xu, and Jindong Chen. 2021 · 2021
Later among the works it cites.
Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels
Xiaoyi Zhang, Lilian de Greef, Amanda Swearngin, Samuel White, Kyle Murray, Lisa Yu, Qi Shan, Jeffrey Nichols, Jason Wu, Chris Fleizach, Aaron Everitt, and Jeffrey P Bigham. 2021 · 2021
Later among the works it cites.