2022

META-GUI: Towards Multi-modal Conversational Agents on Mobile GUI

Sun, Liangtai, Chen, Xingyu, Chen, Lu et al.

Understand

Task-oriented dialogue (TOD) systems have been widely used by mobile phone intelligent assistants to accomplish tasks such as calendar scheduling or hotel reservation.

Current TOD systems usually focus on multi-turn text/speech interaction, then they would call back-end APIs designed for TODs to perform the task.
However, this API-based architecture greatly limits the information-searching capability of intelligent assistants and may even lead to task failure if TOD-specific APIs are not available or the task is too complicated to be executed by the provided APIs.
In this paper, we propose a new TOD architecture: GUI-based task-oriented dialogue system (GUI-TOD).

Built on

Ctrl: A conditional transformer language model for controllable generation
Original
Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. 2019 · 1909
Earlier work this paper cites.
Cued standard dialogue acts
Steve Young. 2007 · 2007
Earlier work this paper cites.
Cognitive technology in task-oriented dialogue systems: Concepts, advances and future
Kai Yu, Lu Chen, Bo Chen, Kai Sun, and Su Zhu. 2014 · 2014
Earlier work this paper cites.
Faster r-cnn: Towards real-time object detection with region proposal networks
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015 · 2015
Earlier work this paper cites.
A survey on dialogue systems: Recent advances and new frontiers
Hongshen Chen, Xiaorui Liu, Dawei Yin, and Jiliang Tang. 2017 · 2017
Earlier work this paper cites.
Sugilite: creating multimodal smartphone automation by demonstration
Toby Jia-Jun Li, Amos Azaria, and Brad A Myers. 2017 · 2017
Earlier work this paper cites.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017 · 2017
Earlier work this paper cites.
Policy adaptation for deep reinforcement learning-based dialogue management
Lu Chen, Cheng Chang, Zhi Chen, Bowen Tan, Milica Gašić, and Kai Yu. 2018 · 2018
Earlier work this paper cites.
Appinite: A multi-modal interface for specifying data descriptions in programming by demonstration using natural language instructions
Toby Jia-Jun Li, Igor Labutov, Xiaohan Nancy Li, Xiaoyi Zhang, Wenze Shi, Wanling Ding, Tom M Mitchell, and Brad A Myers. 2018 · 2018
Earlier work this paper cites.
Kite: Building conversational bots from mobile apps
Toby Jia-Jun Li and Oriana Riva. 2018 · 2018
Earlier work this paper cites.
Mapping natural language commands to web elements
Panupong Pasupat, Tian-Shun Jiang, Evan Zheran Liu, Kelvin Guu, and Percy Liang. 2018 · 2018
Earlier work this paper cites.
Agentgraph: Toward universal dialogue management with structured deep reinforcement learning
Lu Chen, Zhi Chen, Bowen Tan, Sishan Long, Milica Gašić, and Kai Yu. 2019 · 2019
Earlier work this paper cites.

Similar

Google is bringing ai assistant duplex to the web
Tech Crunch. 2019 · 2019
Cited alongside, same era.
BERT: Pre-training of deep bidirectional transformers for language understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 · 2019
Cited alongside, same era.
Multi-step reasoning via recurrent dual attention for visual dialog
Zhe Gan, Yu Cheng, Ahmed Kholy, Linjie Li, Jingjing Liu, and Jianfeng Gao. 2019 · 2019
Cited alongside, same era.
Pumice: A multi-modal agent that learns concepts and conditionals from natural language and demonstrations
Toby Jia-Jun Li, Marissa Radensky, Justin Jia, Kirielle Singarajah, Tom M Mitchell, and Brad A Myers. 2019 · 2019
Cited alongside, same era.
History for visual dialog: Do we really need it?
Shubham Agarwal, Trung Bui, Joon-Young Lee, Ioannis Konstas, and Verena Rieser. 2020 · 2020
Cited alongside, same era.
Task-oriented dialogue as dataflow synthesis
Jacob Andreas, John Bufe, David Burkett, Charles Chen, Josh Clausman, Jean Crawford, Kate Crim, Jordan DeLoach, Leah Dorner, Jason Eisner, et al. 2020 · 2020
Cited alongside, same era.
Video-grounded dialogues with pretrained generation language models
Hung Le and Steven C.H. Hoi. 2020 · 2020
Cited alongside, same era.
BiST: Bi-directional spatio-temporal reasoning for video-grounded dialogues
Hung Le, Doyen Sahoo, Nancy Chen, and Steven C.H. Hoi. 2020 · 2020
Cited alongside, same era.
VD-BERT: A Unified Vision and Dialog Transformer with BERT
Yue Wang, Shafiq Joty, Michael Lyu, Irwin King, Caiming Xiong, and Steven C.H. Hoi. 2020 · 2020
Cited alongside, same era.
Layoutlm: Pre-training of text and layout for document image understanding
Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou. 2020 · 2020
Cited alongside, same era.
Recent advances and challenges in task-oriented dialog systems
Zheng Zhang, Ryuichi Takanobu, Qi Zhu, MinLie Huang, and XiaoYan Zhu. 2020 · 2020
Cited alongside, same era.
Schema-guided multi-domain dialogue state tracking with graph attention neural networks
Lu Chen, Boer Lv, Chi Wang, Su Zhu, Bowen Tan, and Kai Yu. 2020a
Cited in the paper.

Then

Efficient context and schema fusion networks for multi-domain dialogue state tracking
Su Zhu, Jieyu Li, Lu Chen, and Kai Yu. 2020 · 2020
Later among the works it cites.
Uibert: Learning generic multimodal representations for ui understanding
Chongyang Bai, Xiaoxue Zang, Ying Xu, Srinivas Sunkara, Abhinav Rastogi, Jindong Chen, and Blaise Agüera y Arcas. 2021 · 2021
Later among the works it cites.
Actionbert: Leveraging user actions for semantic understanding of user interfaces
Zecheng He, Srinivas Sunkara, Xiaoxue Zang, Ying Xu, Lijuan Liu, Nevan Wichers, Gabriel Schubiner, Ruby Lee, and Jindong Chen. 2021 · 2021
Later among the works it cites.
Glider: A reinforcement learning approach to extract ui scripts from websites
Yuanchun Li and Oriana Riva. 2021 · 2021
Later among the works it cites.
Flin: A flexible natural language interface for web navigation
Sahisnu Mazumder and Oriana Riva. 2021 · 2021
Later among the works it cites.
Etna: Harvesting action graphs from websites
Oriana Riva and Jason Kace. 2021 · 2021
Later among the works it cites.
Grounding open-domain instructions to automate web support tasks
Nancy Xu, Sam Masling, Michael Du, Giovanni Campagna, Larry Heck, James Landay, and Monica Lam. 2021a · 2021
Later among the works it cites.
Large-scale modeling of mobile user click behaviors using deep learning
Xin Zhou and Yang Li. 2021 · 2021
Later among the works it cites.
Few-shot nlu with vector projection distance and abstract triangular crf
Su Zhu, Lu Chen, Ruisheng Cao, Zhi Chen, Qingliang Miao, and Kai Yu. 2021 · 2021
Later among the works it cites.
Dfm: Dialogue foundation model for universal large-scale dialogue-oriented task learning
Original
Zhi Chen, Jiabao Ji, Lu Chen, Yuncong Liu, Da Ma, Bei Chen, Mengyue Wu, Su Zhu, Xin Dong, Fujiang Ge, Qingliang Miao, Jian-Guang Lou, and Kai Yu. 2022 · 2022
Closest in time.
Recent advances in deep learning based dialogue systems: A systematic survey
Jinjie Ni, Tom Young, Vlad Pandelea, Fuzhao Xue, and Erik Cambria. 2022 · 2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…