Fetching the paper…
Reading the bibliography…
We present MUG, a novel interactive task for multimodal grounding where a user and an agent work collaboratively on an interface screen.
Shrdlu: A system for dialog
Terry Winograd · 1972
Earlier work this paper cites.
Learning the semantics of words and pictures
Kobus Barnard and David Forsyth · 2001
Earlier work this paper cites.
A model for learning the semantics of pictures
Victor Lavrenko, Raghavan Manmatha, and Jiwoon Jeon · 2003
Earlier work this paper cites.
Flickr30k entities: Collecting Region-to-Phrase correspondences for richer Image-to-Sentence models
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik · 2015
Earlier work this paper cites.
Faster R-CNN: towards real-time object detection with region proposal networks
Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun · 2015
Earlier work this paper cites.
CIDEr: Consensus-based image description evaluation
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh · 2015
Earlier work this paper cites.
Multimodal compact bilinear pooling for visual question answering and visual grounding
Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, and Marcus Rohrbach · 2016
Earlier work this paper cites.
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2016
Earlier work this paper cites.
Modeling context in referring expressions
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg · 2016
Earlier work this paper cites.
Query-guided regression network with context policy for phrase grounding
Kan Chen, Rama Kovvuri, and Ram Nevatia · 2017
Earlier work this paper cites.
Rico: A mobile app dataset for building data-driven design applications
Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar · 2017
Earlier work this paper cites.
Sugilite: Creating multimodal smartphone automation by demonstration
Toby Jia-Jun Li, Amos Azaria, and Brad A. Myers · 2017
Earlier work this paper cites.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin · 2017
Cited alongside, same era.
Mapping natural language commands to web elements
Panupong Pasupat, Tian-Shun Jiang, Evan Liu, Kelvin Guu, and Percy Liang · 2018
Cited alongside, same era.
Situational impairments during mobile interaction
Zhanna Sarsenbayeva · 2018
Cited alongside, same era.
Mattnet: Modular attention network for referring expression comprehension
Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L Berg · 2018
Cited alongside, same era.
Learning to navigate the web
Izzeddin Gur, Ulrich Rueckert, Aleksandra Faust, and Dilek Hakkani-Tur · 2019
Cited alongside, same era.
The PhotoBook dataset: Building common ground through visually-grounded dialogue
Janosch Haber, Tim Baumgärtner, Ece Takmaz, Lieke Gelderloos, Elia Bruni, and Raquel Fernández · 2019
Refer, Reuse, Reduce: Generating Subsequent References in Visual and Conversational Contexts
Ece Takmaz, Mario Giulianelli, Sandro Pezzelle, Arabella Sinclair, and Raquel Fernández · 2020
Later among the works it cites.
Uibert: Learning generic multimodal representations for ui understanding
Chongyang Bai, Xiaoxue Zang, Ying Xu, Srinivas Sunkara, Abhinav Rastogi, Jindong Chen, et al · 2021
Later among the works it cites.
Grounding ‘grounding’ in NLP
Khyathi Raghavi Chandu, Yonatan Bisk, and Alan W Black · 2021
Later among the works it cites.
Decision transformer: Reinforcement learning via sequence modeling
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch · 2021
Later among the works it cites.
Transvg: End-to-end visual grounding with transformers
Jiajun Deng, Zhengyuan Yang, Tianlang Chen, Wengang Zhou, and Houqiang Li · 2021
Later among the works it cites.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Cited alongside, same era.
Executing instructions in situated collaborative interactions
Alane Suhr, Claudia Yan, Jack Schluger, Stanley Yu, Hadi Khader, Marwa Mouallem, Iris Zhang, and Yoav Artzi · 2019
Cited alongside, same era.
An image is worth 16x16 words: Transformers for image recognition at scale
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby · 2020
Cited alongside, same era.
Characterizing the dynamics of learning in repeated reference games
Robert D Hawkins, Michael C Frank, and Noah D Goodman · 2020
Cited alongside, same era.
ActionBert: Leveraging user actions for semantic understanding of user interfaces
Zecheng He, Srinivas Sunkara, Xiaoxue Zang, Ying Xu, Lijuan Liu, Nevan Wichers, Gabriel Schubiner, Ruby Lee, Jindong Chen, and Blaise Agüera y Arcas · 2020
Cited alongside, same era.
Room-across-room: Multilingual vision-and-language navigation with dense spatiotemporal grounding
Alexander Ku, Peter Anderson, Roma Patel, Eugene Ie, and Jason Baldridge · 2020
Cited alongside, same era.
Widget captioning: Generating natural language description for mobile user interface elements
Yang Li, Gang Li, Luheng He, Jingjie Zheng, Hong Li, and Zhiwei Guan · 2020
Cited alongside, same era.
Analysis of language change in collaborative instruction following
Anna Effenberger, Rhia Singh, Eva Yan, Alane Suhr, and Yoav Artzi · 2021
Later among the works it cites.
Continual learning for grounded instruction generation by observing human following behavior
Noriyuki Kojima, Alane Suhr, and Yoav Artzi · 2021
Later among the works it cites.
Vut: Versatile ui transformer for multi-modal multi-task user interface modeling
Yang Li, Gang Li, Xin Zhou, Mostafa Dehghani, and Alexey Gritsenko · 2021
Later among the works it cites.
Adapting user interfaces with model-based reinforcement learning
Kashyap Todi, Gilles Bailly, Luis Leiva, and Antti Oulasvirta · 2021
Later among the works it cites.
A dataset for interactive vision language navigation with unknown command feasibility
Andrea Burns, Deniz Arsan, Sanjna Agrawal, Ranjitha Kumar, Kate Saenko, and Bryan A. Plummer · 2022
Closest in time.
Learning to denoise raw mobile UI layouts for improving datasets at scale
Gang Li, Gilles Baechler, Manuel Tragut, and Yang Li · 2022
Closest in time.