2021

ProTo: Program-Guided Transformer for Program-Guided Tasks

Zhao, Zelin, Samel, Karan, Chen, Binghong et al.

Understand

Programs, consisting of semantic and structural information, play an important role in the communication between humans and agents.

Towards learning general program executors to unify perception, reasoning, and decision making, we formulate program-guided tasks which require learning to execute a given program on the observed task specification.
Furthermore, we propose the Program-guided Transformer (ProTo), which integrates both semantic and structural guidance of a program by leveraging cross-attention and masked self-attention to pass messages between the specification and routines in the program.
ProTo executes a program in a learned latent space and enjoys stronger representation ability than previous neural-symbolic approaches.

Built on

Program-guided image manipulators
Original
Jiayuan Mao, Xiuming Zhang, Yikai Li, William T. Freeman, Joshua B. Tenenbaum, and Jiajun Wu · 1909
Earlier work this paper cites.
Control flow analysis
Frances E Allen · 1970
Earlier work this paper cites.
Strips: A new approach to the application of theorem proving to problem solving
Richard E Fikes and Nils J Nilsson · 1971
Earlier work this paper cites.
Combining artificial neural networks and symbolic processing for autonomous robot guidance
Dean A Pomerleau, Jay Gowdy, and Charles E Thorpe · 1991
Earlier work this paper cites.
Grounding symbols in the analog world with neural nets
Stevan Harnad · 1993
Earlier work this paper cites.
Markov games as a framework for multi-agent reinforcement learning
Michael L Littman · 1994
Earlier work this paper cites.
Learning from demonstration
Stefan Schaal et al · 1997
Earlier work this paper cites.
Reinforcement learning with hierarchies of machines
Ronald Parr and Stuart Russell · 1998
Earlier work this paper cites.
Learning to forget: Continual prediction with lstm
Felix A Gers, Jürgen Schmidhuber, and Fred Cummins · 1999
Earlier work this paper cites.
Actor-critic algorithms
Vijay R Konda and John N Tsitsiklis · 2000
Earlier work this paper cites.
A perspective view and survey of meta-learning
Ricardo Vilalta and Youssef Drissi · 2002
Earlier work this paper cites.
Programmable reinforcement learning agents
David Andre and Stuart J Russell · 2003
Earlier work this paper cites.
Recent advances in hierarchical reinforcement learning
Andrew G Barto and Sridhar Mahadevan · 2003
Earlier work this paper cites.
Multi-task reinforcement learning: a hierarchical bayesian approach
Aaron Wilson, Alan Fern, Soumya Ray, and Prasad Tadepalli · 2007
Earlier work this paper cites.
Neural-symbolic cognitive reasoning
Artur SD’Avila Garcez, Luis C Lamb, and Dov M Gabbay · 2008
Earlier work this paper cites.
A survey of robot learning from demonstration
Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning · 2009
Earlier work this paper cites.
Learning to execute programs with instruction pointer attention graph neural networks
Original
David Bieber, Charles Sutton, Hugo Larochelle, and Daniel Tarlow · 2010
Earlier work this paper cites.
Learning to interpret natural language navigation instructions from observations
David Chen and Raymond Mooney · 2011
Earlier work this paper cites.
Robot learning from demonstration by constructing skill trees
George Konidaris, Scott Kuindersma, Roderic Grupen, and Andrew Barto · 2012
Earlier work this paper cites.
Max-deeplab: End-to-end panoptic segmentation with mask transformers
Original
Huiyu Wang, Yukun Zhu, Hartwig Adam, Alan L. Yuille, and Liang-Chieh Chen · 2012
Earlier work this paper cites.
An introduction to human-computer interaction (psychology revivals)
Paul Booth · 2014
Earlier work this paper cites.
Path planning with modified a star algorithm for a mobile robot
František Duchoň, Andrej Babinec, Martin Kajan, Peter Beňo, Martin Florek, Tomáš Fico, and Ladislav Jurišica · 2014
Earlier work this paper cites.
Neural turing machines
Original
Alex Graves, Greg Wayne, and Ivo Danihelka · 2014
Earlier work this paper cites.
Program synthesis
Sumit Gulwani · 2014
Earlier work this paper cites.
Vqa: Visual question answering
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh · 2015
Earlier work this paper cites.
Neural gpus learn algorithms
Original
Łukasz Kaiser and Ilya Sutskever · 2015
Earlier work this paper cites.
Human-level concept learning through probabilistic program induction
Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum · 2015
Earlier work this paper cites.
Neural programmer-interpreters
Original
Scott Reed and Nando De Freitas · 2015
Earlier work this paper cites.
Improved semantic representations from tree-structured long short-term memory networks
Original
Kai Sheng Tai, Richard Socher, and Christopher D Manning · 2015
Earlier work this paper cites.

Similar

Simple baseline for visual question answering
Original
Bolei Zhou, Yuandong Tian, Sainbayar Sukhbaatar, Arthur Szlam, and Rob Fergus · 2015
Cited alongside, same era.
Latent attention for if-then program synthesis
Original
Xinyun Chen, Chang Liu, Richard Shin, Dawn Song, and Mingcheng Chen · 2016
Cited alongside, same era.
Differentiable functional program interpreters
Original
John K Feser, Marc Brockschmidt, Alexander L Gaunt, and Daniel Tarlow · 2016
Cited alongside, same era.
Generative adversarial imitation learning
Original
Jonathan Ho and Stefano Ermon · 2016
Cited alongside, same era.
Hierarchical question-image co-attention for visual question answering
Original
Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh · 2016
Cited alongside, same era.
Sentiment analysis: Detecting valence, emotions, and other affectual states from text
Saif M Mohammad · 2016
Cited alongside, same era.
Where to look: Focus regions for visual question answering
Kevin J Shih, Saurabh Singh, and Derek Hoiem · 2016
Cited alongside, same era.
Mastering the game of go with deep neural networks and tree search
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al · 2016
Cited alongside, same era.
Neural-symbolic learning and reasoning: A survey and interpretation
Original
Tarek R. Besold, Artur S. d’Avila Garcez, Sebastian Bader, Howard Bowman, Pedro M. Domingos, Pascal Hitzler, Kai-Uwe Kühnberger, Luís C. Lamb, Daniel Lowd, Priscila Machado Vieira Lima, Leo de Penning, Gadi Pinkas, Hoifung Poon, and Gerson Zaverucha · 2017
Cited alongside, same era.
Towards synthesizing complex programs from input-output examples
Original
Xinyun Chen, Chang Liu, and Dawn Song · 2017
Cited alongside, same era.
Learning to infer graphics programs from hand-drawn images
Original
Kevin Ellis, Daniel Ritchie, Armando Solar-Lezama, and Joshua B Tenenbaum · 2017
Cited alongside, same era.
Visual genome: Connecting language and vision using crowdsourced dense image annotations
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al · 2017
Cited alongside, same era.

Then

Hierarchical reinforcement learning for zero-shot generalization with subtask dependencies
Original
Sungryull Sohn, Junhyuk Oh, and Honglak Lee · 2018
Later among the works it cites.
Mask-guided contrastive attention model for person re-identification
Chunfeng Song, Yan Huang, Wanli Ouyang, and Liang Wang · 2018
Later among the works it cites.
Neural program synthesis from diverse demonstration videos
Shao-Hua Sun, Hyeonwoo Noh, Sriram Somasundaram, and Joseph Lim · 2018
Later among the works it cites.
Programmatically interpretable reinforcement learning
Abhinav Verma, Vijayaraghavan Murali, Rishabh Singh, Pushmeet Kohli, and Swarat Chaudhuri · 2018
Later among the works it cites.
Improving the universality and learnability of neural programmer-interpreters with combinator abstraction
Original
Da Xiao, Jo-Yu Liao, and Xingyuan Yuan · 2018
Later among the works it cites.
Neural-symbolic vqa: Disentangling reasoning from vision and language understanding
Original
Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Joshua B Tenenbaum · 2018
Later among the works it cites.
Neural symbolic reader: Scalable integration of distributed and symbolic representations for reading comprehension
Xinyun Chen, Chen Liang, Adams Wei Yu, Denny Zhou, Dawn Song, and Quoc V Le · 2019
Later among the works it cites.
Neural-symbolic computing: An effective methodology for principled integration of machine learning and reasoning
Original
Artur d’Avila Garcez, Marco Gori, Luis C Lamb, Luciano Serafini, Michael Spranger, and Son N Tran · 2019
Later among the works it cites.
Unicoder: A universal language encoder by pre-training with multiple cross-lingual tasks
Original
Haoyang Huang, Yaobo Liang, Nan Duan, Ming Gong, Linjun Shou, Daxin Jiang, and Ming Zhou · 2019
Later among the works it cites.
Perceptual visual reasoning with knowledge propagation
Guohao Li, Xin Wang, and Wenwu Zhu · 2019
Later among the works it cites.
Learning to describe scenes with programs
Yunchao Liu and Zheng Wu · 2019
Later among the works it cites.
Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks
Original
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee · 2019
Later among the works it cites.
Language models are unsupervised multitask learners
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever · 2019
Later among the works it cites.
Lxmert: Learning cross-modality encoder representations from transformers
Hao Tan and Mohit Bansal · 2019
Later among the works it cites.
Learning to infer and execute 3d shape programs
Original
Yonglong Tian, Andrew Luo, Xingyuan Sun, Kevin Ellis, William T Freeman, Joshua B Tenenbaum, and Jiajun Wu · 2019
Later among the works it cites.
Xlnet: Generalized autoregressive pretraining for language understanding
Original
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le · 2019
Later among the works it cites.
Large batch optimization for deep learning: Training bert in 76 minutes
Original
Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh · 2019
Later among the works it cites.
End-to-end object detection with transformers
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko · 2020
Later among the works it cites.
Compositional generalization via neural-symbolic stack machines
Original
Xinyun Chen, Chen Liang, Adams Wei Yu, Dawn Song, and Denny Zhou · 2020
Later among the works it cites.
An image is worth 16x16 words: Transformers for image recognition at scale
Original
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al · 2020
Later among the works it cites.
One policy to control them all: Shared modular policies for agent-agnostic control
Wenlong Huang, Igor Mordatch, and Deepak Pathak · 2020
Later among the works it cites.
Object-centric diagnosis of visual reasoning
Original
Jianwei Yang, Jiayuan Mao, Jiajun Wu, Devi Parikh, David D Cox, Joshua B Tenenbaum, and Chuang Gan · 2020
Later among the works it cites.
Augmenting policy learning with routines discovered from a demonstration
Original
Zelin Zhao, Chuang Gan, Jiajun Wu, Xiaoxiao Guo, and Joshua Tenenbaum · 2020
Later among the works it cites.
Meta module network for compositional visual reasoning
Wenhu Chen, Zhe Gan, Linjie Li, Yu Cheng, William Wang, and Jingjing Liu · 2021
Closest in time.
Mask attention networks: Rethinking and strengthen transformer
Original
Zhihao Fan, Yeyun Gong, Dayiheng Liu, Zhongyu Wei, Siyuan Wang, Jian Jiao, Nan Duan, Ruofei Zhang, and Xuanjing Huang · 2021
Closest in time.
Swin transformer: Hierarchical vision transformer using shifted windows
Original
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo · 2021
Closest in time.
Learning to synthesize programs as interpretable and generalizable policies, 2021
Dweep Trivedi, Jesse Zhang, Shao-Hua Sun, and Joseph J. Lim · 2021
Closest in time.
Rest: An efficient transformer for visual recognition
Original
QingLong Zhang and Yubin Yang · 2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…