2020

ActionBert: Leveraging User Actions for Semantic Understanding of User Interfaces

He, Zecheng, Sunkara, Srinivas, Zang, Xiaoxue et al.

Understand

As mobile devices are becoming ubiquitous, regularly interacting with a variety of user interfaces (UIs) is a common aspect of daily life for many people.

To improve the accessibility of these devices and to enable their usage in a variety of settings, building models that can assist users and accomplish tasks through the UI is vitally important.
However, there are several challenges to achieve this.
First, UI components of similar appearance can have different functionalities, making understanding their function more important than just analyzing their appearance.

Built on

Fusion of Detected Objects in Text for Visual Question Answering
Original
Alberti, C.; Ling, J.; Collins, M.; and Reitter, D. 2019 · 1908
Earlier work this paper cites.
Visualbert: A Simple and Performant Baseline for Vision and Language
Original
Li, L. H.; Yatskar, M.; Yin, D.; Hsieh, C.-J.; and Chang, K.-W. 2019 · 1908
Earlier work this paper cites.
Imagebert: Cross-modal Pre-training with Large-scale Weak-supervised Image-text Data
Original
Qi, D.; Su, L.; Song, J.; Cui, E.; Bharti, T.; and Sacheti, A. 2020 · 2001
Earlier work this paper cites.
Understanding the Difficulty of Training Deep Feedforward Neural Networks
Glorot, X.; and Bengio, Y. 2010 · 2010
Earlier work this paper cites.
Adam: A Method for Stochastic Optimization
Original
Kingma, D. P.; and Ba, J. 2014 · 2014
Earlier work this paper cites.
Faster R-CNN: Towards Real-time Object Detection with Region Proposal Networks
Ren, S.; He, K.; Girshick, R.; and Sun, J. 2015 · 2015
Earlier work this paper cites.
Layer Normalization
Original
Ba, J. L.; Kiros, J. R.; and Hinton, G. E. 2016 · 2016
Earlier work this paper cites.
Deep Residual Learning for Image Recognition
He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016 · 2016
Earlier work this paper cites.

Similar

Rico: A Mobile App Dataset for Building Data-driven Design Applications
Deka, B.; Huang, Z.; Franzen, C.; Hibschman, J.; Afergan, D.; Li, Y.; Nichols, J.; and Kumar, R. 2017 · 2017
Cited alongside, same era.
Attention Is All You Need
Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017 · 2017
Cited alongside, same era.
Universal Sentence Encoder
Original
Cer, D.; Yang, Y.; Kong, S.-y.; Hua, N.; Limtiaco, N.; John, R. S.; Constant, N.; Guajardo-Cespedes, M.; Yuan, S.; Tar, C.; et al. 2018 · 2018
Cited alongside, same era.
Bert: Pre-training of Deep Bidirectional Transformers for Language Understanding
Original
Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2018 · 2018
Cited alongside, same era.
Learning Design Semantics for Mobile Apps
Liu, T. F.; Craft, M.; Situ, J.; Yumer, E.; Mech, R.; and Kumar, R. 2018 · 2018
Cited alongside, same era.
Resolving Referring Expressions in Images with Labeled Elements
Wichers, N.; Hakkani-Tür, D.; and Chen, J. 2018 · 2018
Cited alongside, same era.
Swire: Sketch-Based User Interface Retrieval
Huang, F.; Canny, J. F.; and Nichols, J. 2019 · 2019
Cited alongside, same era.

Then

Vilbert: Pretraining Task-agnostic Visiolinguistic Representations for Vision-and-language Tasks
Lu, J.; Batra, D.; Parikh, D.; and Lee, S. 2019 · 2019
Later among the works it cites.
VL-BERT: Pre-training of Generic Visual-Linguistic Representations
Su, W.; Zhu, X.; Cao, Y.; Li, B.; Lu, L.; Wei, F.; and Dai, J. 2019 · 2019
Later among the works it cites.
Modeling Mobile Interface Tappability Using Crowdsourcing and Deep Learning
Swearngin, A.; and Li, Y. 2019 · 2019
Later among the works it cites.
Understanding and Modeling User-Perceived Brand Personality from Mobile Application UIs
Wu, Z.; Kim, T.; Li, Q.; and Ma, X. 2019 · 2019
Later among the works it cites.
Uniter: Universal Image-text Representation Learning
Chen, Y.-C.; Li, L.; Yu, L.; Kholy, A. E.; Ahmed, F.; Gan, Z.; Cheng, Y.; and Liu, J. 2020 · 2020
Closest in time.
Robo App Crawler Documentation
Firebase. 2020 · 2020
Closest in time.
Predicting and Diagnosing User Engagement with Mobile UI Animation via a Data-Driven Approach
Wu, Z.; Jiang, Y.; Liu, Y.; and Ma, X. 2020 · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…