2022

RelViT: Concept-guided Vision Transformer for Visual Relational Reasoning

Ma, Xiaojian, Nie, Weili, Yu, Zhiding et al.

Understand

Reasoning about visual relationships is central to how humans interpret the visual world.

This task remains challenging for current deep learning algorithms since it requires addressing three key technical problems jointly: 1) identifying object entities and their properties, 2) inferring semantic relations between pairs of entities, and 3) generalizing to novel object-relation combinations, i.e., systematic generalization.
In this work, we use vision transformers (ViTs) as our base model for visual reasoning and make better use of concepts defined as object entities and their relations to improve the reasoning ability of ViTs.
Specifically, we introduce a novel concept-feature dictionary to allow flexible image feature retrieval at training time with concept keys.

Built on

Hake: Human activity knowledge engine
Original
Yong-Lu Li, Liang Xu, Xinpeng Liu, Xijie Huang, Yue Xu, Mingyang Chen, Ze Ma, Shiyi Wang, Hao-Shu Fang, and Cewu Lu · 1904
Earlier work this paper cites.
Visualbert: A simple and performant baseline for vision and language
Original
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang · 1908
Earlier work this paper cites.
A stochastic grammar of images
Song-Chun Zhu and David Mumford · 2007
Earlier work this paper cites.
Sift flow: Dense correspondence across scenes and its applications
Ce Liu, Jenny Yuen, and Antonio Torralba · 2010
Earlier work this paper cites.
Phrasal recognition
Ali Farhadi and Mohammad Amin Sadeghi · 2013
Earlier work this paper cites.
HICO: A benchmark for recognizing human-object interactions in images
Yu-Wei Chao, Zhan Wang, Yugeng He, Jiaxuan Wang, and Jia Deng · 2015
Earlier work this paper cites.
Faster r-cnn: Towards real-time object detection with region proposal networks
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun · 2015
Earlier work this paper cites.
Attend, infer, repeat: Fast scene understanding with generative models
SM Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Geoffrey E Hinton, et al · 2016
Earlier work this paper cites.
beta-vae: Learning basic visual concepts with a constrained variational framework
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner · 2016
Earlier work this paper cites.
Visual genome: Connecting language and vision using crowdsourced dense image annotations
Original
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, Michael Bernstein, and Li Fei-Fei · 2016
Earlier work this paper cites.
Learning models for actions and person-object interactions with transfer to question answering
Arun Mallya and Svetlana Lazebnik · 2016
Earlier work this paper cites.
Attentional pooling for action recognition
Original
Rohit Girdhar and Deva Ramanan · 2017
Earlier work this paper cites.
Clevr: A diagnostic dataset for compositional language and elementary visual reasoning
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick · 2017
Earlier work this paper cites.
Shapeworld-a new test methodology for multimodal language understanding
Original
Alexander Kuhnle and Ann Copestake · 2017
Earlier work this paper cites.
Transitive invariance for self-supervised visual representation learning
Xiaolong Wang, Kaiming He, and Abhinav Gupta · 2017
Earlier work this paper cites.
Bottom-up and top-down attention for image captioning and visual question answering
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang · 2018
Earlier work this paper cites.
Systematic generalization: what is required and can it be learned?
Original
Dzmitry Bahdanau, Shikhar Murty, Michael Noukhovitch, Thien Huu Nguyen, Harm de Vries, and Aaron Courville · 2018
Earlier work this paper cites.
Measuring abstract reasoning in neural networks
David Barrett, Felix Hill, Adam Santoro, Ari Morcos, and Timothy Lillicrap · 2018
Earlier work this paper cites.
Deep clustering for unsupervised learning of visual features
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze · 2018
Earlier work this paper cites.
Pairwise body-part attention for recognizing human-object interactions
Hao-Shu Fang, Jinkun Cao, Yu-Wing Tai, and Cewu Lu · 2018
Earlier work this paper cites.

Similar

Disentangling by factorising
Hyunjik Kim and Andriy Mnih · 2018
Cited alongside, same era.
Flexible neural representation for physics prediction
Original
Damian Mrowca, Chengxu Zhuang, Elias Wang, Nick Haber, Li Fei-Fei, Joshua B Tenenbaum, and Daniel LK Yamins · 2018
Cited alongside, same era.
Representation learning with contrastive predictive coding
Original
Aaron van den Oord, Yazhe Li, and Oriol Vinyals · 2018
Cited alongside, same era.
Scaling human-object interaction recognition through zero-shot learning
Liyue Shen, Serena Yeung, Judy Hoffman, Greg Mori, and Li Fei-Fei · 2018
Cited alongside, same era.
Unsupervised feature learning via non-parametric instance discrimination
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin · 2018
Cited alongside, same era.
Neural-symbolic vqa: Disentangling reasoning from vision and language understanding
Original
Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Joshua B Tenenbaum · 2018
Cited alongside, same era.
Multi-object representation learning with iterative variational inference
Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner · 2019
Cited alongside, same era.
Gqa: A new dataset for real-world visual reasoning and compositional question answering
Drew A Hudson and Christopher D Manning · 2019
Cited alongside, same era.
Challenging common assumptions in the unsupervised learning of disentangled representations
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem · 2019
Cited alongside, same era.
Lxmert: Learning cross-modality encoder representations from transformers
Original
Hao Tan and Mohit Bansal · 2019
Cited alongside, same era.
Learning correspondence from the cycle-consistency of time
Xiaolong Wang, Allan Jabri, and Alexei A Efros · 2019
Cited alongside, same era.
Deep modular co-attention networks for visual question answering
Zhou Yu, Jun Yu, Yuhao Cui, Dacheng Tao, and Qi Tian · 2019
Cited alongside, same era.

Then

In defense of grid features for visual question answering
Huaizu Jiang, Ishan Misra, Marcus Rohrbach, Erik Learned-Miller, and Xinlei Chen · 2020
Later among the works it cites.
Concept bottleneck models
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang · 2020
Later among the works it cites.
Space: Unsupervised object-oriented scene representation via spatial attention and decomposition
Original
Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn · 2020
Later among the works it cites.
Object-centric learning with slot attention
Original
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf · 2020
Later among the works it cites.
Bongard-logo: A new benchmark for human-level concept learning and reasoning
Weili Nie, Zhiding Yu, Lei Mao, Ankit B Patel, Yuke Zhu, and Anima Anandkumar · 2020
Later among the works it cites.
A benchmark for systematic generalization in grounded language understanding
Original
Laura Ruis, Jacob Andreas, Marco Baroni, Diane Bouchacourt, and Brenden M Lake · 2020
Later among the works it cites.
V-prom: A benchmark for visual reasoning using visual progressive matrices
Damien Teney, Peng Wang, Jiewei Cao, Lingqiao Liu, Chunhua Shen, and Anton van den Hengel · 2020
Later among the works it cites.
Contrastive multiview coding
Yonglong Tian, Dilip Krishnan, and Phillip Isola · 2020
Later among the works it cites.
Emerging properties in self-supervised vision transformers
Original
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin · 2021
Later among the works it cites.
Exploring simple siamese representation learning
Xinlei Chen and Kaiming He · 2021
Later among the works it cites.
An empirical study of training self-supervised vision transformers
Original
Xinlei Chen, Saining Xie, and Kaiming He · 2021
Later among the works it cites.
Improve vision transformers training by suppressing over-smoothing
Original
Chengyue Gong, Dilin Wang, Meng Li, Vikas Chandra, and Qiang Liu · 2021
Later among the works it cites.
Is object detection necessary for human-object interaction recognition?
Original
Ying Jin, Yinpeng Chen, Lijuan Wang, Jianfeng Wang, Pei Yu, Zicheng Liu, and Jenq-Neng Hwang · 2021
Later among the works it cites.
Efficient self-supervised vision transformers for representation learning
Original
Chunyuan Li, Jianwei Yang, Pengchuan Zhang, Mei Gao, Bin Xiao, Xiyang Dai, Lu Yuan, and Jianfeng Gao · 2021
Later among the works it cites.
Swin transformer: Hierarchical vision transformer using shifted windows
Original
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo · 2021
Later among the works it cites.
Halma: Humanlike abstraction learning meets affordance in rapid problem solving
Original
Sirui Xie, Xiaojian Ma, Peiyu Yu, Yixin Zhu, Ying Nian Wu, and Song-Chun Zhu · 2021
Later among the works it cites.
Unsupervised foreground extraction via deep region competition
Peiyu Yu, Sirui Xie, Xiaojian Ma, Yixin Zhu, Ying Nian Wu, and Song-Chun Zhu · 2021
Later among the works it cites.
Scaling vision transformers
Original
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer · 2021
Later among the works it cites.
Bongard-hoi: Benchmarking few-shot visual reasoning for human-object interactions
Huaizu Jiang, Xiaojian Ma, Weili Nie, Zhiding Yu, Yuke Zhu, and Anima Anandkumar · 2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…