Fetching the paper…
Reading the bibliography…
Mobile app user interfaces (UIs) are rich with action, text, structure, and image content that can be utilized to learn generic UI representations for tasks like automating user commands, summarizing content, and evaluating the accessibility of user interfaces.
A suite of generative tasks for multi-level multimodal webpage understanding
Andrea Burns, Krishna Srinivasan, Joshua Ainslie, Geoff Brown, Bryan Plummer, Kate Saenko, Jianmo Ni, and Mandy Guo. 2023 · 1947
Earlier work this paper cites.
The Use of the Internet for Educational Purposes
Nazan Dogruer, Ramadan Eyyam, and Ipek Menevis. 2011 · 2011
Earlier work this paper cites.
Privacy concerns and behaviors of people with visual impairments
Tousif Ahmed, Roberto Hoyle, Kay Connelly, David Crandall, and Apu Kapadia. 2015 · 2015
Earlier work this paper cites.
CIDEr: Consensus-based Image Description Evaluation
Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015 · 2015
Earlier work this paper cites.
Show and tell: A neural image caption generator
Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015 · 2015
Earlier work this paper cites.
Discovering Different Kinds of Smartphone Users through Their Application Usage Behaviors
Sha Zhao, Julian Ramos, Jianrong Tao, Ziwen Jiang, Shijian Li, Zhaohui Wu, Gang Pan, and Anind K. Dey. 2016 · 2016
Earlier work this paper cites.
Stochastic variational video prediction
Mohammad Babaeizadeh, Chelsea Finn, Dumitru Erhan, Roy H. Campbell, and Sergey Levine. 2017 · 2017
Earlier work this paper cites.
Deep visual foresight for planning robot motion
Chelsea Finn and Sergey Levine. 2017 · 2017
Earlier work this paper cites.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017 · 2017
Earlier work this paper cites.
Stochastic adversarial video prediction
Alex X. Lee, Richard Zhang, Frederik Ebert, Pieter Abbeel, Chelsea Finn, and Sergey Levine. 2018 · 2018
Earlier work this paper cites.
Visual reinforcement learning with imagined goals
Ashvin V Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. 2018 · 2018
Earlier work this paper cites.
Look before you leap: Bridging model-free and model-based reinforcement learning for planned-ahead vision-and-language navigation
Xin Wang, Wenhan Xiong, Hongmin Wang, and William Wang. 2018 · 2018
Earlier work this paper cites.
Hierarchical foresight: Self-supervised learning of long-horizon tasks via visual subgoal generation
Suraj Nair and Chelsea Finn. 2019 · 2019
Earlier work this paper cites.
Modeling mobile interface tappability using crowdsourcing and deep learning
Amanda Swearngin and Yang Li. 2019 · 2019
Earlier work this paper cites.
Bridging screen readers and voice assistants for enhanced eyes-free web search
Alexandra Vtyurina, Adam Fourney, Meredith Ringel Morris, Leah Findlater, and Ryen W. White. 2019 · 2019
Earlier work this paper cites.
“i am uncomfortable sharing what i can’t see”: privacy concerns of the visually impaired with camera based assistive applications
Taslima Akter, Bryan Dosono, Tousif Ahmed, Apu Kapadia, and Bryan Semaan. 2020 · 2020
Cited alongside, same era.
Visuospatial foresight for multi-step, multi-task fabric manipulation
Ryan Hoque, Daniel Seita, Ashwin Balakrishna, Aditya Ganapathi, Ajay Kumar Tanwani, Nawid Jamali, Katsu Yamane, Soshi Iba, and Ken Goldberg. 2020 · 2020
Cited alongside, same era.
Widget captioning: Generating natural language description for mobile user interface elements
Yang Li, Gang Li, Luheng He, Jingjie Zheng, Hong Li, and Zhiwei Guan. 2020 · 2020
Cited alongside, same era.
Goal-aware prediction: Learning to model what matters
Suraj Nair, Silvio Savarese, and Chelsea Finn. 2020 · 2020
Cited alongside, same era.
Exploring the limits of transfer learning with a unified text-to-text transformer
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020 · 2020
Cited alongside, same era.
Wit: Wikipedia-based image text dataset for multimodal multilingual machine learning
Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Marc Najork. 2021 · 2021
Later among the works it cites.
Screen2words: Automatic mobile ui summarization with multimodal learning
Bryan Wang, Gang Li, Xin Zhou, Zhourong Chen, Tovi Grossman, and Yang Li. 2021 · 2021
Later among the works it cites.
A dataset for interactive vision language navigation with unknown command feasibility
Andrea Burns, Deniz Arsan, Sanjna Agrawal, Ranjitha Kumar, Kate Saenko, and Bryan A. Plummer. 2022 · 2022
Later among the works it cites.
Scaling instruction-finetuned language models
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022 · 2022
Later among the works it cites.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Bleurt: Learning robust metrics for text generation
Thibault Sellam, Dipanjan Das, and Ankur P Parikh. 2020 · 2020
Cited alongside, same era.
Experience-embedded visual foresight
Lin Yen-Chen, Maria Bauza, and Phillip Isola. 2020 · 2020
Cited alongside, same era.
Uibert: Learning generic multimodal representations for ui understanding
Chongyang Bai, Xiaoxue Zang, Ying Xu, Srinivas Sunkara, Abhinav Rastogi, Jindong Chen, and Blaise Aguera y Arcas. 2021 · 2021
Cited alongside, same era.
VirTex: Learning Visual Representations from Textual Annotations
Karan Desai and Justin Johnson. 2021 · 2021
Cited alongside, same era.
An image is worth 16x16 words: Transformers for image recognition at scale
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021 · 2021
Cited alongside, same era.
A fine-grained analysis of BERTScore
Michael Hanna and Ondřej Bojar. 2021 · 2021
Cited alongside, same era.
Actionbert: Leveraging user actions for semantic understanding of user interfaces
Zecheng He, Srinivas Sunkara, Xiaoxue Zang, Ying Xu, Lijuan Liu, Nevan Wichers, Gabriel Schubiner, Ruby Lee, and Jindong Chen. 2021 · 2021
Cited alongside, same era.
A large-scale longitudinal analysis of missing label accessibility failures in android apps
Raymond Fok, Mingyuan Zhong, Anne Spencer Ross, James Fogarty, and Jacob O. Wobbrock. 2022 · 2022
Later among the works it cites.
Learning to denoise raw mobile ui layouts for improving datasets at scale
Gang Li, Gilles Baechler, Manuel Tragut, and Yang Li. 2022a · 2022
Later among the works it cites.
Tl;dw? summarizing instructional videos with task relevance & cross-modal saliency
Medhini G. Narasimhan, Arsha Nagrani, Chen Sun, Michael Rubinstein, Trevor Darrell, Anna Rohrbach, and Cordelia Schmid. 2022 · 2022
Later among the works it cites.
ChatGPT
OpenAI. 2022 · 2022
Later among the works it cites.
Predicting and explaining mobile ui tappability with vision modeling and saliency analysis
Eldon Schoop, Xin Zhou, Gang Li, Zhourong Chen, Bjoern Hartmann, and Yang Li. 2022 · 2022
Later among the works it cites.
Opt: Open pre-trained transformer language models
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022 · 2022
Later among the works it cites.
Instructblip: Towards general-purpose vision-language models with instruction tuning
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023 · 2023
Later among the works it cites.
Spotlight: Mobile ui understanding using vision-language models with a focus
Gang Li and Yang Li. 2023 · 2023
Later among the works it cites.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023 · 2023
Later among the works it cites.
Android in the wild: A large-scale dataset for android device control
Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. 2023 · 2023
Later among the works it cites.
Visualwebarena: Evaluating multimodal agents on realistic visual web tasks
Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. 2024 · 2024
Closest in time.