2018

Machine Learning-Based Prototyping of Graphical User Interfaces for Mobile Apps

Moran, Kevin, Bernal-Cárdenas, Carlos, Curcio, Michael et al.

Understand

It is common practice for developers of user-facing software to transform a mock-up of a graphical user interface (GUI) into code.

This process takes place both at an application's inception and in an evolutionary context as GUI changes keep pace with evolving features.
Unfortunately, this practice is challenging and time-consuming.
In this paper, we present an approach that automates this process by enabling accurate prototyping of GUIs via three tasks: detection, classification, and assembly.

Built on

W. S. Lasecki, J. Kim, N. Rafter, O. Sen, J. P. Bigham, and M. S. Bernstein, “Apparition: Crowdsourced user interfaces that come to life as you sketch them,” in Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems , ser. CHI ’15. New York, NY, USA: ACM, 2015, pp. 1925–1934. [Online]. Available: http://doi.acm.org/10.1145/2702123.2702565
1934
Earlier work this paper cites.
J. Canny, “A computational approach to edge detection,” IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. PAMI-8, no. 6, pp. 679–698, Nov 1986
1986
Earlier work this paper cites.
B. Myers, “Challenges of hci design and implementation,” Interactions , vol. 1, no. 1, pp. 73–83, Jan. 1994. [Online]. Available: http://doi.acm.org.proxy.wm.edu/10.1145/174800.174808
1994
Earlier work this paper cites.
J. A. Landay and B. A. Myers, “Interactive sketching for the early stages of user interface design,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems , ser. CHI ’95. New York, NY, USA: ACM Press/Addison-Wesley Publishing Co., 1995, pp. 43–50. [Online]. Available: http://dx.doi.org/10.1145/223904.223910
1995
Earlier work this paper cites.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” Proceedings of the IEEE , vol. 86, no. 11, pp. 2278–2324, Nov 1998
1998
Earlier work this paper cites.
W. Conover, Practical Nonparametric Statistics . Wiley, 1998
1998
Earlier work this paper cites.
J. A. Landay and B. A. Myers, “Sketching interfaces: toward more human interface design,” Computer , vol. 34, no. 3, pp. 56–64, Mar 2001
2001
Earlier work this paper cites.
A. Caetano, N. Goulart, M. Fonseca, and J. Jorge, “Javasketchit: Issues in sketching the look of user interfaces,” in AAAI Spring Symposium on Sketch Understanding , ser. SSS’02, 2002, pp. 9–14
2002
Earlier work this paper cites.
A. B. Tucker, Computer Science Handbook, Second Edition . Chapman & Hall/CRC, 2004
2004
Earlier work this paper cites.
S. Chatty, S. Sire, J.-L. Vinot, P. Lecoanet, A. Lemort, and C. Mertz, “Revisiting visual interface programming: Creating gui tools for designers and programmers,” in Proceedings of the 17th Annual ACM Symposium on User Interface Software and Technology , ser. UIST ’04. New York, NY, USA: ACM, 2004, pp. 267–276. [Online]. Available: http://doi.acm.org/10.1145/1029632.1029678
2004
Earlier work this paper cites.
G. Csurka, C. Dance, L. Fan, J. Willamowski, and C. Bray, “Visual categorization with bags of keypoints,” in Workshop on statistical learning in computer vision , ser. ECCV’04, vol. 1, no. 1-22. Prague, 2004, pp. 1–2
2004
Earlier work this paper cites.
R. J. Grissom and J. J. Kim, Effect sizes for research: A broad practical approach . Lawrence Earlbaum Associates, 2005
2005
Earlier work this paper cites.
A. Coyette, S. Kieffer, and J. Vanderdonckt, “Multi-fidelity prototyping of user interfaces,” in Proceedings of the 11th IFIP TC 13 International Conference on Human-computer Interaction , ser. INTERACT’07. Berlin, Heidelberg: Springer-Verlag, 2007, pp. 150–164. [Online]. Available: http://dl.acm.org/citation.cfm?id=1776994.1777015
2007
Earlier work this paper cites.
H. Samir and A. Kamel, “Automated reverse engineering of java graphical user interfaces for web migration,” in 2007 ITI 5th International Conference on Information and Communications Technology , ser. ICICT’07, Dec 2007, pp. 157–162
2007
Earlier work this paper cites.
H. Bay, A. Ess, T. Tuytelaars, and L. Van Gool, “Speeded-up robust features (surf),” Comput. Vis. Image Underst. , vol. 110, no. 3, pp. 346–359, Jun. 2008. [Online]. Available: http://dx.doi.org/10.1016/j.cviu.2007.09.014
2007
Earlier work this paper cites.
B. Myers, S. Y. Park, Y. Nakano, G. Mueller, and A. Ko, “How designers design and program interactive behaviors,” in 2008 IEEE Symposium on Visual Languages and Human-Centric Computing , Sept 2008, pp. 177–184
2008
Earlier work this paper cites.
M. Grechanik, Q. Xie, and C. Fu, “Creating gui testing tools using accessibility technologies,” in 2009 International Conference on Software Testing, Verification, and Validation Workshops , ser. ICSTW’09, April 2009, pp. 243–250
2009
Earlier work this paper cites.
M. Dixon and J. Fogarty, “Prefab: Implementing advanced behaviors using pixel-based reverse engineering of interface structure,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems , ser. CHI ’10. New York, NY, USA: ACM, 2010, pp. 1525–1534. [Online]. Available: http://doi.acm.org/10.1145/1753326.1753554
2010
Earlier work this paper cites.
A. Hinze, J. Bowen, Y. Wang, and R. Malik, “Model-driven gui & interaction design using emulation,” in Proceedings of the 2Nd ACM SIGCHI Symposium on Engineering Interactive Computing Systems , ser. EICS ’10. New York, NY, USA: ACM, 2010, pp. 273–278. [Online]. Available: http://doi.acm.org/10.1145/1822018.1822061
2010
Earlier work this paper cites.
S. Roy Choudhary, H. Versee, and A. Orso, “Webdiff: Automated identification of cross-browser issues in web applications,” in Proceedings of the 2010 IEEE International Conference on Software Maintenance , ser. ICSM ’10. Washington, DC, USA: IEEE Computer Society, 2010, pp. 1–10. [Online]. Available: http://dx.doi.org/10.1109/ICSM.2010.5609723
2010
Earlier work this paper cites.
J. Seifert, B. Pfleging, E. del Carmen Valderrama Bahamóndez, M. Hermes, E. Rukzio, and A. Schmidt, “Mobidev: A tool for creating apps on mobile phones,” in Proceedings of the 13th International Conference on Human Computer Interaction with Mobile Devices and Services (MobileHCI’11) , ser. MobileHCI ’11. New York, NY, USA: ACM, 2011, pp. 109–112. [Online]. Available: http://doi.acm.org/10.1145/2037373.2037392
2011
Earlier work this paper cites.
T.-H. Chang, T. Yeh, and R. Miller, “Associating the visual representation of user interfaces with their internal structures and metadata,” in Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology , ser. UIST ’11. New York, NY, USA: ACM, 2011, pp. 245–256. [Online]. Available: http://doi.acm.org/10.1145/2047196.2047228
2011
Earlier work this paper cites.

Similar

M. Dixon, D. Leventhal, and J. Fogarty, “Content and hierarchy in pixel-based methods for reverse engineering interface structure,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems , ser. CHI ’11. New York, NY, USA: ACM, 2011, pp. 969–978. [Online]. Available: http://doi.acm.org/10.1145/1978942.1979086
2011
Cited alongside, same era.
E. Shah and E. Tilevich, “Reverse-engineering user interfaces to facilitate porting to and across mobile devices and platforms,” in Proceedings of the Compilation of the Co-located Workshops on DSM’11, TMC’11, AGERE! 2011, AOOPES’11, NEAT’11, & VMIL’11 , ser. SPLASH ’11 Workshops. New York, NY, USA: ACM, 2011, pp. 255–260. [Online]. Available: http://doi.acm.org/10.1145/2095050.2095093
2011
Cited alongside, same era.
A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Advances in Neural Information Processing Systems 25 , F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, Eds. Curran Associates, Inc., 2012, pp. 1097–1105. [Online]. Available: http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf
2012
Cited alongside, same era.
S. Anand, M. Naik, M. J. Harrold, and H. Yang, “Automated concolic testing of smartphone apps,” in Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering , ser. FSE’12. New York, NY, USA: ACM, 2012, pp. 59:1–59:11. [Online]. Available: http://doi.acm.org/10.1145/2393596.2393666
2012
Cited alongside, same era.
D. Amalfitano, A. R. Fasolino, P. Tramontana, S. De Carmine, and A. M. Memon, “Using gui ripping for automated testing of android applications,” in Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering , ser. ASE’12. New York, NY, USA: ACM, 2012, pp. 258–261. [Online]. Available: http://doi.acm.org/10.1145/2351676.2351717
2012
Cited alongside, same era.
S. R. Choudhary, M. R. Prasad, and A. Orso, “Crosscheck: Combining crawling and differencing to better detect cross-browser incompatibilities in web applications,” in Proceedings of the 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation , ser. ICST ’12. Washington, DC, USA: IEEE Computer Society, 2012, pp. 171–180. [Online]. Available: http://dx.doi.org/10.1109/ICST.2012.97
2012
Cited alongside, same era.
C. Zeidler, C. Lutteroth, W. Stuerzlinger, and G. Weber, Evaluating Direct Manipulation Operations for Constraint-Based Layout . Berlin, Heidelberg: Springer Berlin Heidelberg, 2013, pp. 513–529. [Online]. Available: http://dx.doi.org/10.1007/978-3-642-40480-1_35
2013
Cited alongside, same era.
A. Machiry, R. Tahiliani, and M. Naik, “Dynodroid: An input generation system for android apps,” in Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering , ser. ESEC/FSE’13. New York, NY, USA: ACM, 2013, pp. 224–234. [Online]. Available: http://doi.acm.org/10.1145/2491411.2491450
2013
Cited alongside, same era.
H. Ye, S. Cheng, L. Zhang, and F. Jiang, “Droidfuzzer: Fuzzing the android apps with intent-filter tag,” in Proceedings of International Conference on Advances in Mobile Computing & Multimedia , ser. MoMM ’13. New York, NY, USA: ACM, 2013, pp. 68:68–68:74. [Online]. Available: http://doi.acm.org/10.1145/2536853.2536881
2013
Cited alongside, same era.
T. Azim and I. Neamtiu, “Targeted and depth-first exploration for systematic testing of android apps,” in Proceedings of the 2013 ACM SIGPLAN International Conference on Object Oriented Programming Systems Languages & Applications , ser. OOPSLA ’13. New York, NY, USA: ACM, 2013, pp. 641–660. [Online]. Available: http://doi.acm.org/10.1145/2509136.2509549
2013
Cited alongside, same era.
W. Yang, M. R. Prasad, and T. Xie, “A grey-box approach for automated gui-model generation of mobile applications,” in Proceedings of the 16th International Conference on Fundamental Approaches to Software Engineering , ser. FASE’13. Berlin, Heidelberg: Springer-Verlag, 2013, pp. 250–265. [Online]. Available: http://dx.doi.org/10.1007/978-3-642-37057-1_19
2013
Cited alongside, same era.
W. Choi, G. Necula, and K. Sen, “Guided gui testing of android apps with minimal restart and approximate learning,” in Proceedings of the 2013 ACM SIGPLAN International Conference on Object Oriented Programming Systems Languages & Applications , ser. OOPSLA ’13. New York, NY, USA: ACM, 2013, pp. 623–640. [Online]. Available: http://doi.acm.org/10.1145/2509136.2509552
2013
Cited alongside, same era.

Then

I. M. Ruiz, M. Nagappan, B. Adams, T. Berger, S. Dienst, and A. Hassan, “On the relationship between the number of ad libraries in an android app and its rating,” IEEE Software , no. 1, pp. 1–1, 2014
2014
Later among the works it cites.
R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature hierarchies for accurate object detection and semantic segmentation,” in Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition , ser. CVPR ’14. Washington, DC, USA: IEEE Computer Society, 2014, pp. 580–587. [Online]. Available: http://dx.doi.org/10.1109/CVPR.2014.81
2014
Later among the works it cites.
T. A. Nguyen and C. Csallner, “Reverse engineering mobile application user interfaces with remaui,” in Proceedings of the 2015 30th IEEE/ACM International Conference on Automated Software Engineering , ser. ASE’15. Washington, DC, USA: IEEE Computer Society, 2015, pp. 248–259. [Online]. Available: http://dx.doi.org/10.1109/ASE.2015.32
2015
Later among the works it cites.
V. Lelli, A. Blouin, and B. Baudry, “Classifying and qualifying gui defects,” in 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST) , April 2015, pp. 1–10
2015
Later among the works it cites.
C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in Computer Vision and Pattern Recognition (CVPR) , 2015. [Online]. Available: http://arxiv.org/abs/1409.4842
Original
2015
Later among the works it cites.
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei, “Imagenet large scale visual recognition challenge,” Int. J. Comput. Vision , vol. 115, no. 3, pp. 211–252, Dec. 2015. [Online]. Available: http://dx.doi.org/10.1007/s11263-015-0816-y
2015
Later among the works it cites.
M. Linares-Vásquez, M. White, C. Bernal-Cárdenas, K. Moran, and D. Poshyvanyk, “Mining android app usages for generating actionable gui-based execution scenarios,” in 12th Working Conference on Mining Software Repositories (MSR’15) , 2015, p. to appear
2015
Later among the works it cites.
K. Moran, M. Linares-Vásquez, C. Bernal-Cárdenas, and D. Poshyvanyk, “Auto-completing bug reports for android applications,” in in Proceedings of 10th Joint Meeting of the European Software Engineering Conference and the 23rd ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/FSE’15) , Bergamo, Italy, Ausgust-September 2015 2015, p. to appear
2015
Later among the works it cites.
S. R. Choudhary, A. Gorla, and A. Orso, “Automated test input generation for android: Are we there yet?” in 30th IEEE/ACM International Conference on Automated Software Engineering (ASE 2015) , 2015
2015
Later among the works it cites.
J. Gui, S. Mcilroy, M. Nagappan, and W. G. Halfond, “Truth in advertising: The hidden cost of mobile ads for software developers,” in Proceedings of the 37th International Conference on Software Engineering , ser. ICSE’15. Florence, Italy: IEEE Press, 2015, pp. 100–110
2015
Later among the works it cites.
Z. Wang, X. Wang, and G. Wang, “Learning fine-grained features via a CNN tree for large-scale classification,” CoRR , vol. abs/1511.04534, 2015. [Online]. Available: http://arxiv.org/abs/1511.04534
Original
2015
Later among the works it cites.
S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” in Proceedings of the 28th International Conference on Neural Information Processing Systems , ser. NIPS’15. Cambridge, MA, USA: MIT Press, 2015, pp. 91–99. [Online]. Available: http://dl.acm.org/citation.cfm?id=2969239.2969250
2015
Later among the works it cites.
K. He, X. Zhang, S. Ren, and J. Sun, in 2016 IEEE Conference on Computer Vision and Pattern Recognition , ser. CVPR’16
2016
Later among the works it cites.
K. Moran, M. Linares-Vásquez, C. Bernal-Cárdenas, C. Vendome, and D. Poshyvanyk, “Automatically discovering, reporting and reproducing android application crashes,” in Proceedings of the IEEE International Conference on Software Testing, Verification and Validation (ICST’16) . IEEE, 2016, pp. 33–44
2016
Later among the works it cites.
U. Karn, “An intuitive explanation of convolutional neural nets https://ujjwalkarn.me/2016/08/11/intuitive-explanation-convnets/ .”
2016
Later among the works it cites.
K. Mao, M. Harman, and Y. Jia, “Sapienz: Multi-objective automated testing for android applications,” in Proceedings of the 25th International Symposium on Software Testing and Analysis , ser. ISSTA’16. New York, NY, USA: ACM, 2016, pp. 94–105. [Online]. Available: http://doi.acm.org/10.1145/2931037.2931054
2016
Later among the works it cites.
T. Beltramelli, “pix2code: Generating code from a graphical user interface screenshot,” CoRR , vol. abs/1705.07962, 2017. [Online]. Available: http://arxiv.org/abs/1705.07962
Original
2017
Later among the works it cites.
B. Deka, Z. Huang, C. Franzen, J. Hibschman, D. Afergan, Y. Li, J. Nichols, and R. Kumar, “Rico: A mobile app dataset for building data-driven design applications,” in Proceedings of the 30th Annual Symposium on User Interface Software and Technology , ser. UIST ’17, 2017
2017
Later among the works it cites.
K. Moran, M. Linares-Vásquez, C. Bernal-Cárdenas, C. Vendome, and D. Poshyvanyk, “Crashscope: A practical tool for automated testing of android applications,” in Proceedings of the 39th International Conference on Software Engineering Companion , ser. ICSE-C ’17. Piscataway, NJ, USA: IEEE Press, 2017, pp. 15–18. [Online]. Available: https://doi.org/10.1109/ICSE-C.2017.16
2017
Later among the works it cites.
W. Fu and T. Menzies, “Easy over hard: A case study on deep learning,” in Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering , ser. ESEC/FSE’17. New York, NY, USA: ACM, 2017, pp. 49–60. [Online]. Available: http://doi.acm.org/10.1145/3106237.3106256
2017
Later among the works it cites.
K. Moran, B. Li, C. Bernal-Cárdenas, D. Jelf, and D. Poshyvanyk, “Automated reporting of gui design violations in mobile apps,” in Proceedings of the 40th International Conference on Software Engineering Companion , ser. ICSE ’18. Piscataway, NJ, USA: IEEE Press, 2018, p. to appear
2018
Closest in time.
“Opencv https://opencv.org .”
2018
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…